Ask Your Question
0

Unable to ping VM from an external host through its floating ip

asked 2015-05-11 00:22:53 -0600

harshalx gravatar image

updated 2015-05-23 02:46:48 -0600

Hello, I am trying to install Juno on an Amazon VM with RHEL 7. My installation goes perfectly without a hitch. I am also able to access the Dash board, create an external and internal network and start a VM. However, I am stuck at a point where I cannot access this VM that I create.

After reading up a lot I understood that an OVS bridge needs to be configured. I followed the instructions given at the link below point by point.

Neutron_with_existing_external_network

At the point when I restart the 'network' service my AWS instance loses connectivity with the network completely. I am left with no other alternative but to terminate this instance and start the RDO installation, over a new instance, all over again. Losing connectivity also means that I don't have the chance to get back to the machine and analyze the neutron or openvswitch logs to figure out what went wrong.

Here is what my cofigurations look like:

/etc/sysconfig/network-scripts/ifcfg-br-ex

DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=10.3.3.143 # Old eth0 IP that is assigned by AWS to eth0
NETMASK=255.255.255.0  # your netmask
GATEWAY=10.3.3.1  # your gateway
DNS1=10.3.2.234     # your nameserver
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0 
HWADDR=52:54:00:92:05:AE # This is the hwaddr of eth0 that is originally given by AWS
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=br-ex
ONBOOT=yes

 /etc/neutron/plugin.ini

network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-ex

neutron agent-list shows that all agents are running fine and dandy with :) against each.

I know that its not idea to install Openstack over a VM. However, I really don't have the resources to get a bare metal server and hence this is my only option.

Can anybody please guide me further to understand where things might be going wrong?

UPDATE

/etc/sysconfig/network-scripts/ifcfg-br-ex

DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=10.3.3.219_br_ex
NETMASK=255.255.255.0_br_ex
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
ONBOOT=yes
BOOTPROTO=none

[root@ip-10-3-3-219 ~(keystone_admin)]# ovs-vsctl show
0e11330a-f3de-4b8e-aa5d-01c1a4183d7c
    Bridge br-int
        fail_mode: secure
        Port "tap9e51a51f-1f"
            tag: 1
            Interface "tap9e51a51f-1f"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qvo770ee127-d1"
            tag: 1
            Interface "qvo770ee127-d1"
        Port br-int
            Interface br-int
                type: internal
        Port "qr-c3690b5d-e9"
            tag: 1
            Interface "qr-c3690b5d-e9"
                type: internal
    Bridge br-ex
        Port "qg-a83a7152-29"
            Interface "qg-a83a7152-29"
                type: internal
        Port "eth0"
            Interface "eth0"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.3.1"

[root@ip-10-3-3-219 ~(keystone_admin)]# ifconfig
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.3.3.219  netmask 255.255.255.0  broadcast 10.3.3.255
        inet6 fe80::4f3:38ff:fec7:be59  prefixlen 64  scopeid 0x20<link>
        ether 06:f3:38:c7:be:59  txqueuelen 0  (Ethernet)
        RX packets 84577  bytes 67169262 (64.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame ...
(more)
edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted
0

answered 2015-05-16 00:23:06 -0600

updated 2015-05-17 00:35:45 -0600

Due to known issue with "G+" logins https://bugs.launchpad.net/openstack-...
I cannot update my original feed and use another one to respond

I presume , that you have performed RDO Juno AIO install via `packstack --allinone`, then
packstack by default performs ML2&OVS&VXLAN setup on single host

# cat ifcfg-br-ex
DEVICE="br-ex"
BOOTPROTO="static"
IPADDR="10.3.3.219"
NETMASK="255.255.255.0"
DNS1="8.8.8.8"
BROADCAST="10.3.3.255"
GATEWAY="10.3.3.1"  <== should be real gateway's IP on office LAN
NM_CONTROLLED="no"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="yes"
IPV6INIT=no
ONBOOT="yes"
TYPE="OVSIntPort"
OVS_BRIDGE=br-ex
DEVICETYPE="ovs"

Your plugin.ini should look like :-

[root@aio-hostname neutron(keystone_admin)]# ls -l
total 72
-rw-r--r--. 1 root root      197 May 11 17:07 api-paste.ini
drwxr-xr-x. 8 root root     4096 May 11 17:06 conf.d
-rw-r-----. 1 root neutron  3605 Apr 30 15:52 dhcp_agent.ini
-rw-r-----. 1 root neutron  5035 Apr 30 15:52 l3_agent.ini
-rw-r-----. 1 root neutron  2512 Apr 30 15:52 metadata_agent.ini
-rw-r-----. 1 root neutron 36245 May 11 17:07 neutron.conf
lrwxrwxrwx. 1 root root       37 May 11 17:06 plugin.ini -> /etc/neutron/plugins/ml2/ml2_conf.ini
drwxr-xr-x. 3 root root       16 May 11 17:06 plugins
-rw-r-----. 1 root neutron  7140 Apr 30 15:52 policy.json
-rw-r--r--. 1 root root     1164 Apr 30 15:52 rootwrap.conf
[root@aio-hostname neutron(keystone_admin)]# cat plugin.ini | grep -v ^#| grep -v ^$
[ml2]
type_drivers = vxlan
tenant_network_types = vxlan
mechanism_drivers =openvswitch
[ml2_type_flat]
[ml2_type_vlan]
[ml2_type_gre]
[ml2_type_vxlan]
vni_ranges =1001:2000
vxlan_group =239.1.1.2
[securitygroup]
enable_security_group = True

$ source keystonerc_demo
$ nova secgroup-list-rules default

+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
edit flag offensive delete link more

Comments

Thanks for the reply dbaxps. Remember, my all-in-one installation wasn't working and hence I followed your blog and set it up via the answer file? After I did that the whole installation worked fine, but I still have the problem that I cannot access the floating ip of the vm from outside.

harshalx gravatar imageharshalx ( 2015-05-16 15:10:11 -0600 )edit

Have implemented security rules ? View updated answer

dbaxps-Second-Incarnation gravatar imagedbaxps-Second-Incarnation ( 2015-05-17 00:36:50 -0600 )edit

Next step. During pinging floating IP of VM identify tap interface plugged by OVS ( via brctl show) && run tcpdump -vv -i tap-interface. Check would you see ICMP replies from VM on tap plugin. Identify qvo-interface (@br-int) corresponding tap related qvb-interface

dbaxps gravatar imagedbaxps ( 2015-05-17 02:47:40 -0600 )edit

Run tcpdump -vv -i qvo-interface. Then respond to thread

dbaxps gravatar imagedbaxps ( 2015-05-17 02:48:58 -0600 )edit
0

answered 2015-05-13 03:25:28 -0600

harshalx gravatar image

Thanks for the replies dbaxps. I followed your blog and was successfully able to get the ovs bridge configuration up. However, I am still having issues accessing my vm. Here are some more details:

 [root@ip-10-3-3-219 ~]# ip netns
qrouter-396c7174-ce92-4b89-ba05-ffc1afc9a1ca
qdhcp-f3d8fdfe-c6e0-4ba6-a2f3-12b57f09d635
qrouter-cdb38450-a1ee-4024-b85e-02af405d4901

[root@ip-10-3-3-219 ~]# ip netns exec qrouter-396c7174-ce92-4b89-ba05-ffc1afc9a1ca iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 10.3.3.249/32 -j DNAT --to-destination 192.168.1.2
-A neutron-l3-agent-POSTROUTING ! -i qg-8a044353-c1 ! -o qg-8a044353-c1 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 10.3.3.249/32 -j DNAT --to-destination 192.168.1.2
-A neutron-l3-agent-float-snat -s 192.168.1.2/32 -j SNAT --to-source 10.3.3.249
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -s 192.168.1.0/24 -j SNAT --to-source 10.3.3.251
-A neutron-postrouting-bottom -j neutron-l3-agent-snat

My VM got allocated his ip - 192.168.1.2 I associated a floating ip with it - 10.3.3.249. This ip belongs to my external network subnet.

However, when I ping 10.3.3.249 or ssh into it .. there is a connection timeout. What might be going wrong here?

edit flag offensive delete link more

Comments

Post , your ifcfg-br-ex, ifcfg-eth0 ( or whatever converted to OVS port), ovs-vsctl show && ifconfig.

dbaxps-Second-Incarnation gravatar imagedbaxps-Second-Incarnation ( 2015-05-13 12:34:51 -0600 )edit

dbaxps, The thing 2 note is that I have reached to a point where I am able to ping my private ip and my public ip from the all-in-one host. I am also able to ssh into this VM. However, I am still wondering why my floating ip cannot be pinged from other hosts on the 10.3.3.0/24 network. Any pointers?

harshalx gravatar imageharshalx ( 2015-05-15 09:37:45 -0600 )edit

My question is posted right above yours last request. If you want to get an answer post as UPDATE to your question

your ifcfg-br-ex, ifcfg-eth0 ( or whatever converted to OVS port), ovs-vsctl show && ifconfig.
dbaxps-Second-Incarnation gravatar imagedbaxps-Second-Incarnation ( 2015-05-15 10:32:19 -0600 )edit

Thanks dbaxps. I've posted the contents of these files in an update.

harshalx gravatar imageharshalx ( 2015-05-15 21:42:12 -0600 )edit
0

answered 2015-05-11 01:26:56 -0600

dbaxps gravatar image

updated 2015-05-20 08:33:38 -0600

Next step. During pinging floating IP of VM identify tap interface plugged by OVS ( via brctl show) && run tcpdump -vv -i tap-interface. Check would you see ICMP replies from VM on tap plugin. Identify qvo-interface (@br-int) corresponding tap related qvb-interface

TCPDUMP Troubleshooting on Compute Node :-

    [root@ip-192-169-142-137 ~]# brctl show
    bridge name bridge id       STP enabled interfaces
    qbr7e94450a-6a      8000.daaf111690c4   no      qvb7e94450a-6a
                                tap7e94450a-6a
    qbrcd244411-ad      8000.7ad4b8e2896b   no      qvbcd244411-ad
    [root@ip-192-169-142-137 ~]# ovs-vsctl show | grep 7e94450a-6a
            Port "qvo7e94450a-6a"
                Interface "qvo7e94450a-6a"
    [root@ip-192-169-142-137 ~]# 

# tcpdump -vv -i tap7e94450a-6a
# tcpdump -vv -i qvo7e94450a-6a

Another check to be done , if you implemented security rules via neutron(nova) CLI or via Horizion, make sure corresponding Neutron chains, actually , making this rules to work are present in /etc/sysconfig/iptables :-

[root@ip-192-169-142-137 ~]# iptables-save | grep 7e94450a-6a
-A neutron-openvswi-FORWARD -m physdev --physdev-out tap7e94450a-6a --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-in tap7e94450a-6a --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-openvswi-sg-chain
-A neutron-openvswi-INPUT -m physdev --physdev-in tap7e94450a-6a --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-openvswi-o7e94450a-6
-A neutron-openvswi-sg-chain -m physdev --physdev-out tap7e94450a-6a --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-i7e94450a-6
-A neutron-openvswi-sg-chain -m physdev --physdev-in tap7e94450a-6a --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-o7e94450a-6

192.168.1.0/24 is mgmt && external network

[root@CentOS71WKS ~(keystone_admin)]# route -n
Kernel IP routing table
Destination     Gateway         Genmask     Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0        UG    0      0    0  br-ex
169.254.0.0     0.0.0.0     255.255.0.0       U    1002   0   0 enp2s0  <== Metadata access
169.254.0.0     0.0.0.0     255.255.0.0       U     1005   0   0 br-ex <==  Metadata access 
192.168.1.0     0.0.0.0    255.255.255.0      U     0      0   0 br-ex

View also :- http://bderzhavets.blogspot.com/2014/...

edit flag offensive delete link more

Comments

Thanks for your reply dbaxps. I did run packstack --allinone and dropped all routers, networks, subnets and namespaces and created new as per my network requirements as soon as it was done.

If network_vlan_ranges = physnet1 is incorrect then what should it be?

harshalx gravatar imageharshalx ( 2015-05-11 03:46:08 -0600 )edit

In case of RDO Juno AIO setup it is ML2&OVS&VXLAN setup don't touch ml2_conf.ini at all. No entries like

network_vlan_ranges = physnet

In case you specifically configure answer-file for ML2&OVS&VLAN setup , please , view the link I posted in the answer.

dbaxps gravatar imagedbaxps ( 2015-05-11 03:58:41 -0600 )edit

Please find my replies and the trace at : ICMP Tracing I can see the ping requests and responses from tap to qvo interfaces. Let me know, if, for some reason you cannot see the pastebin link.

harshalx gravatar imageharshalx ( 2015-05-18 18:24:20 -0600 )edit

OK. tcpdump -vv -i br-ex during pinging public IP of your VM from outside. br-ex should be on the same network. Do ICMP replies from VM arrive on br-ex ?

dbaxps gravatar imagedbaxps ( 2015-05-19 01:26:44 -0600 )edit

No. I verified that. When I ping 10.3.6.3 from an outside host on the same network (for e.g 10.3.6.247) the ICMP pings dont reach br-ex. Does that point to a problem with the external network itself,

harshalx gravatar imageharshalx ( 2015-05-19 01:45:05 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2015-05-11 00:22:53 -0600

Seen: 3,722 times

Last updated: May 23 '15