Ask Your Question
0

facing networking issue with VM instance on distributed deployment - Havana release

asked 2014-01-14 09:13:18 -0600

shridhar gravatar image

updated 2014-01-14 09:15:53 -0600

We have installed openstack Havana Version with multi-node configuration.

This deployment is done using 3 seperate network as suggested by neutron config ( Network topology diagram attached)

  • All etho interfaces on 1 network

    • controller eth1/network node bridge interface and glance eth1 is on external network

    • Nova/compute/networknode on 2nd network

Components installed as -

Controller Node -

      Ubuntu 12.04.3, (Keystone,Nova services, Neutrol service, Horizon etc), eth0 on 172.21.1.2 and eth1 172.24.1.2

Glance Node -

      Ubuntu 12.04.3, (glance service ), eth0 on 172.21.1.4 and eth1 172.24.1.4

Network Node -

      Ubuntu 12.04.3, (Neutron service ), eth0 on 172.21.1.3, eth1 172.22.1.2 eth2 bridge interface - 172.24.1.3

Compute Node (KVM) -

      Ubuntu 12.04.3,  eth0 on 172.21.1.101, eth1 172.22.1.101

VM inastancenetworking - private network - 172.23.21.x Floating IP - 172.24.21.x

Problem sequence -

Step1- Created external network with 172.24.0.0/16 with admin project (shared)

Step2- Created internal network with 172.23.0.0/16 with admin project (shared)

Step3- Created soft-router in dashboard for admin project with gateway as 172.24.0.1

Step4- Added internal network as private interface

Step5- Launch new VM instance from the dashboard (success)

Step6- Observed no IP assgined to newly created VM, hence assigned internal IP 172.23.21.2

Step7- Assgin floating IP to this VM. Successfulyl showing it is assigned with 172.24.21.2

Step8- From VM instance tried to ping gateway or any other network address of all 3 networks ( FAILED)*

We are not sure, If our network toplogy is not correct or we are hitting some kind known/unknown issue

Any help/guidence to resolve the problem will be highly appreaciated.

We are attaching Network topology diagram attached as reference.

Thanks

edit retag flag offensive close merge delete

Comments

We are geeting errors - to upload you need >10 karmas.. Any other way to upload while I am accumulating karmas?

shridhar gravatar imageshridhar ( 2014-01-14 09:19:41 -0600 )edit

any error logs etc that you can supply might help

Rajesh Kanade gravatar imageRajesh Kanade ( 2014-01-14 12:49:31 -0600 )edit

You can post the api, compute and neutron logs in pastebin. I can help you to troubleshoot this issue.

dheeru gravatar imagedheeru ( 2014-01-15 01:55:31 -0600 )edit

2 answers

Sort by ยป oldest newest most voted
0

answered 2014-01-15 01:08:47 -0600

shalmali gravatar image

updated 2014-01-16 06:29:43 -0600

Following is the Deployment diagram for the same. Posting on behalf of Shridhar. Network-topology-diagram.jpg

Attaching logs

/var/log/openvswitch/ovs-vswitchd.log (Network node)

2014-01-16T11:50:39Z|00051|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2014-01-16T11:50:39Z|00052|bridge|INFO|bridge br-tun: using datapath ID 00001ef18ac7254b 2014-01-16T11:50:39Z|00053|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" 2014-01-16T11:50:39Z|00054|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device patch-tun failed: No such device 2014-01-16T11:50:39Z|00055|dpif|WARN|system@ovs-system: failed to add patch-tun as port: No such device 2014-01-16T11:50:39Z|00056|netdev_vport|ERR|patch-tun: patch type requires valid 'peer' argument 2014-01-16T11:50:39Z|00057|bridge|WARN|could not configure network device patch-tun (Invalid argument) 2014-01-16T11:50:39Z|00058|bridge|INFO|bridge br-int: added interface patch-tun on port 3 2014-01-16T11:50:39Z|00059|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device patch-int failed: No such device 2014-01-16T11:50:39Z|00060|dpif|WARN|system@ovs-system: failed to add patch-int as port: No such device 2014-01-16T11:50:39Z|00061|netdev_vport|ERR|patch-int: patch type requires valid 'peer' argument 2014-01-16T11:50:39Z|00062|bridge|WARN|could not configure network device patch-int (Invalid argument) 2014-01-16T11:50:39Z|00063|bridge|INFO|bridge br-tun: added interface patch-int on port 1 2014-01-16T11:50:40Z|00064|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device gre-2 failed: No such device 2014-01-16T11:50:40Z|00065|dpif|WARN|system@ovs-system: failed to add gre-2 as port: No such device 2014-01-16T11:50:40Z|00066|netdev_vport|ERR|gre-2: gre type requires valid 'remote_ip' argument 2014-01-16T11:50:40Z|00067|bridge|WARN|could not configure network device gre-2 (Invalid argument) 2014-01-16T11:50:40Z|00068|bridge|INFO|bridge br-tun: added interface gre-2 on port 2 2014-01-16T11:50:43Z|00069|bridge|INFO|bridge br-int: added interface qr-021a5d17-d0 on port 4 2014-01-16T11:50:44Z|00070|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-021a5d17-d0 device failed: No such device 2014-01-16T11:50:44Z|00071|bridge|INFO|bridge br-ex: added interface qg-f43c42c9-94 on port 3 2014-01-16T11:50:44Z|00072|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qr-021a5d17-d0 device failed: No such device 2014-01-16T11:50:44Z|00073|netdev_linux|WARN|qr-021a5d17-d0: removing policing failed: No such device 2014-01-16T11:50:44Z|00074|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-f43c42c9-94 device failed: No such device 2014-01-16T11:50:44Z|00075|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qg-f43c42c9-94 device failed: No such device 2014-01-16T11:50:44Z|00076|netdev_linux|WARN|qg-f43c42c9-94: removing policing failed: No such device 2014-01-16T11:50:45Z|00077|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2014-01-16T11:50:45Z|00078|bridge|INFO|bridge br-tun: using datapath ID 000042b6ef888e4f 2014-01-16T11:50:45Z|00079|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" 2014-01-16T11:50:45Z|00080|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device patch-tun failed: No such device 2014-01-16T11:50:45Z|00081|dpif|WARN|system@ovs-system: failed to add patch-tun as port: No such device 2014-01-16T11:50:45Z|00082|netdev_vport|ERR|patch-tun: patch type requires valid 'peer' argument 2014-01-16T11:50:45Z|00083|bridge|WARN ... (more)

edit flag offensive delete link more
0

answered 2014-02-05 17:44:08 -0600

Andrew Kinney gravatar image

updated 2014-02-05 17:50:55 -0600

We're experiencing the same issue. Also Havana. Our all of our nodes are debian wheezy and the virtualization host is XenServer.

I believe this is key:

"removing policing failed: No such device"

If you look at the device names it complains about, those are the device names associated with the virtual router. It doesn't show in your logs, but our logs indicate the same is happening with the tap interface for the DHCP server defined by OpenStack.

When OpenVSwitch first sets up the virtual router related ports, it inserts flows set to "actions=drop". Those have a higher priority than the VLAN decoding flow that gets inserted later. Thus, the frames are dropped even before VLAN tag actions are taken. It's hosed at layer 2, so nothing works at layer 3 or higher.

At first, I thought that it was because our VM resides on a XenServer compute node and the "root_helper = neutron-rootwrap-xen-dom0 /etc/neutron/rootwrap.conf" statement in /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini was erroneously intercepting calls to OpenVSwitch that should have been directed to the network node instead. I haven't yet ruled that out.

However, another possibility occurred to me. Maybe the OpenVSwitch plugin hasn't properly implemented the network namespaces support. Those device names don't exist in the default network name space. They only exist in the network namespaces created by neutron. Maybe the command to clean up the "action=drop" flow rules in OpenVSwitch was not run in the proper network namespace.

I'm consider ditching or changing the "firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver" statement in /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini. Doing firewalling within OpenVSwitch seems broken, at least in the sense that once it sets up "drop" flows for the ports, it can't remove them.

Comments?

edit flag offensive delete link more

Comments

Every start of the service "neutron-openvswitch-agent" on the network node rebuilds the "drop" flows seen with 'ovs-ofctl dump-flows [br-int|br-ex|br-eth2]' on the port numbers where the connecting ports between br-int and other bridges are built (int-br-ex, int-br-eth2, phy-br-ex, and phy-br-eth2).

Andrew Kinney gravatar imageAndrew Kinney ( 2014-02-05 19:46:24 -0600 )edit

When starting the neutron-l3-agent or the neutron-dhcp-agent on the network node, the neutron-plugin-openvswitch-agent does properly remove the "drop" flows for the virtual interfaces created by those agents. It just doesn't handle the connection from the data bridge (br-eth2 for us) to br-int.

Andrew Kinney gravatar imageAndrew Kinney ( 2014-02-05 20:32:06 -0600 )edit

Adding/editing "ovs_use_veth = True" in both /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini solved the messages about "removing policing failed: No such device" showing in the logs. Still, the issue of the connection from the data bridge to br-int with "actions=drop" flows remains.

Andrew Kinney gravatar imageAndrew Kinney ( 2014-02-05 20:57:12 -0600 )edit

Hard rebooting all the nodes resolved my isuue.

shalmali gravatar imageshalmali ( 2014-02-07 01:33:07 -0600 )edit

Using a VM image with VLAN tags preprogrammed into the networking config of the VM gets this working. Seems like a hack, though, and I'm probably missing something.

Andrew Kinney gravatar imageAndrew Kinney ( 2014-02-11 15:41:40 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

6 followers

Stats

Asked: 2014-01-14 09:13:18 -0600

Seen: 2,325 times

Last updated: Feb 05 '14