Ask Your Question
0

Quantum: When using OVS, virtual interfaces silently disappear or lose their IPs

asked 2012-11-15 22:06:12 -0500

martin-loschwitz gravatar image

Hello all,

I'm running an OpenStack setup here with Quantum+OVS. As I run all services on one node, I need use_namespaces=true in my configuration file. With that, I can set up Quantum & OpenvSwitch nicely, but as soon as I fire up a VM, all the interfaces created by Quantum and OpenVSwitch silently disappear:

35: qbr54d531d7-f3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UP link/ether 8e:7d:9b:e9:6d:87 brd ff:ff:ff:ff:ff:ff inet6 fe80::3871:9ff:fef7:16f7/64 scope link valid_lft forever preferred_lft forever 36: qvo54d531d7-f3: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether ea:7f:8f:ca:f7:6c brd ff:ff:ff:ff:ff:ff inet6 fe80::e87f:8fff:feca:f76c/64 scope link valid_lft forever preferred_lft forever 37: qvb54d531d7-f3: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast master qbr54d531d7-f3 state UP qlen 1000 link/ether 8e:7d:9b:e9:6d:87 brd ff:ff:ff:ff:ff:ff inet6 fe80::8c7d:9bff:fee9:6d87/64 scope link valid_lft forever preferred_lft forever 38: qbrc004d01b-cf: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UP link/ether a2:08:7e:7a:5e:47 brd ff:ff:ff:ff:ff:ff inet6 fe80::c072:a6ff:fea7:b450/64 scope link valid_lft forever preferred_lft forever 39: qvoc004d01b-cf: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether f6:28:36:fc:8a:c0 brd ff:ff:ff:ff:ff:ff inet6 fe80::f428:36ff:fefc:8ac0/64 scope link valid_lft forever preferred_lft forever 40: qvbc004d01b-cf: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast master qbrc004d01b-cf state UP qlen 1000 link/ether a2:08:7e:7a:5e:47 brd ff:ff:ff:ff:ff:ff inet6 fe80::a008:7eff:fe7a:5e47/64 scope link valid_lft forever preferred_lft forever 41: vnet0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast master qbr54d531d7-f3 state UNKNOWN qlen 500 link/ether fe:16:3e:c4:70:a8 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fec4:70a8/64 scope link valid_lft forever preferred_lft forever 42: vnet1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast master qbrc004d01b-cf state UNKNOWN qlen 500 link/ether fe:16:3e:8e:52:dc brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fe8e:52dc/64 scope link valid_lft forever preferred_lft forever

With use_namespaces=False in my configuration, this doesn't happen.

My configuration mainly follows Emilien Macchi's howto; it's using GRE.

I hope anyone can help me on this one!

Best regards Martin G. Loschwitz

edit retag flag offensive close merge delete

16 answers

Sort by ยป oldest newest most voted
0

answered 2012-11-23 13:27:42 -0500

martin-loschwitz gravatar image

Turns out I was just too stupid for this. The interfaces did not "disappear", they were just moved into different network namespaces, exactly the way it is supposed to happen with use_namespaces=true.

edit flag offensive delete link more
0

answered 2012-11-16 01:44:48 -0500

gongysh gravatar image

why cannot I see the vnetxxx at #14 before starting quantum now? The VMs were deleted? are u using fedora or ubuntu? and the version number of kernel and ovs? Thanks

edit flag offensive delete link more
0

answered 2012-11-16 01:22:21 -0500

martin-loschwitz gravatar image

Okay, so let's bring some order into this. Here is what I can see before I start anything related to Quantum (openvswitch is running, though):

root@alice:~# ip a 1: lo: <loopback,up,lower_up> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 169.254.169.254/32 scope link lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:10:ab:99:1c brd ff:ff:ff:ff:ff:ff inet 192.168.122.111/24 brd 192.168.122.255 scope global eth0 inet6 fe80::5054:10ff:feab:991c/64 scope link valid_lft forever preferred_lft forever 3: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:10:cd:a9:74 brd ff:ff:ff:ff:ff:ff inet 192.168.133.111/24 brd 192.168.133.255 scope global eth1 inet6 fe80::5054:10ff:fecd:a974/64 scope link valid_lft forever preferred_lft forever 4: eth2: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:10:ef:a9:74 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:10ff:feef:a974/64 scope link valid_lft forever preferred_lft forever 6: br-int: <broadcast,multicast> mtu 1500 qdisc noop state DOWN link/ether 9e:67:7a:08:10:4d brd ff:ff:ff:ff:ff:ff 7: br-ex: <broadcast,multicast> mtu 1500 qdisc noop state DOWN link/ether 6a:59:e1:58:45:40 brd ff:ff:ff:ff:ff:ff 12: br-tun: <broadcast,multicast> mtu 1500 qdisc noop state DOWN link/ether 5e:6e:e0:5d:f7:41 brd ff:ff:ff:ff:ff:ff

And after starting Quantum, I see this:

root@alice:~# ip a 1: lo: <loopback,up,lower_up> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 169.254.169.254/32 scope link lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:10:ab:99:1c brd ff:ff:ff:ff:ff:ff inet 192.168.122.111/24 brd 192.168.122.255 scope global eth0 inet6 fe80::5054:10ff:feab:991c/64 scope link valid_lft forever preferred_lft forever 3: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:10:cd:a9:74 brd ff:ff:ff:ff:ff:ff inet 192.168.133.111/24 brd 192.168.133.255 scope global eth1 inet6 fe80::5054:10ff:fecd:a974/64 scope link valid_lft forever preferred_lft forever 4: eth2: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast ... (more)

edit flag offensive delete link more
0

answered 2012-11-16 01:10:07 -0500

danwent gravatar image

The devices that appear and then are removed in #11 are created and destroyed by Nova when VMs are booted and destroyed. Are you saying that those specific devices (e.g., vnet0) where destroyed without deleting the VMs? That would be very surprising.

The other devices (e.g., tapXXX and qrYYY or qgwZZZZ) are not supposed to be in the main namespace if use_namespaces=True

what do you see when you run "ip netns list" ? And then for each namespace name you see, run:

ip netns exec <namespace-name> ip addr

edit flag offensive delete link more
0

answered 2012-11-16 01:08:29 -0500

martin-loschwitz gravatar image

Here are some messages that look suspicious -- I see these when firing up the quantum OVS agent:

Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 add-port br-int patch-tun Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-tun type=patch Nov 16 02:07:32 alice ovs-vswitchd: 00035|netdev_vport|ERR|patch-tun: patch type requires valid 'peer' argument Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-tun options:peer=patch-int Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 add-port br-tun patch-int Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-int type=patch Nov 16 02:07:32 alice ovs-vswitchd: 00056|netdev_vport|ERR|patch-int: patch type requires valid 'peer' argument Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-int options:peer=patch-tun

edit flag offensive delete link more
0

answered 2012-11-16 01:03:49 -0500

martin-loschwitz gravatar image

This might be helpful, too (ntp is unrelated but immediately starts to listen on a new interface when it appears and stops to do so upon interface disappearence):

Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 25 qbr54d531d7-f3 fe80::3871:9ff:fef7:16f7 UDP 123 Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 26 qbrc004d01b-cf fe80::c072:a6ff:fea7:b450 UDP 123 Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 27 qvbc004d01b-cf fe80::a008:7eff:fe7a:5e47 UDP 123 Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 28 qvb54d531d7-f3 fe80::8c7d:9bff:fee9:6d87 UDP 123 Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 29 qvo54d531d7-f3 fe80::e87f:8fff:feca:f76c UDP 123 Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 30 qvoc004d01b-cf fe80::f428:36ff:fefc:8ac0 UDP 123 Nov 15 22:58:39 alice ntpd[3148]: peers refreshed Nov 15 22:58:39 alice ntpd[3148]: new interface(s) found: waking up resolver Nov 15 22:58:51 alice ntpd[3148]: Listen normally on 31 vnet1 fe80::fc16:3eff:fe8e:52dc UDP 123 Nov 15 22:58:51 alice ntpd[3148]: Listen normally on 32 vnet0 fe80::fc16:3eff:fec4:70a8 UDP 123 Nov 15 22:58:51 alice ntpd[3148]: peers refreshed Nov 15 22:58:51 alice ntpd[3148]: new interface(s) found: waking up resolver Nov 15 23:09:44 alice ntpd[3148]: Deleting interface #32 vnet0, fe80::fc16:3eff:fec4:70a8#123, interface stats: received=0, sent=0, dropped=0, active_time=653 secs Nov 15 23:09:44 alice ntpd[3148]: Deleting interface #31 vnet1, fe80::fc16:3eff:fe8e:52dc#123, interface stats: received=0, sent=0, dropped=0, active_time=653 secs Nov 15 23:09:44 alice ntpd[3148]: peers refreshed Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #30 qvoc004d01b-cf, fe80::f428:36ff:fefc:8ac0#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #29 qvo54d531d7-f3, fe80::e87f:8fff:feca:f76c#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #28 qvb54d531d7-f3, fe80::8c7d:9bff:fee9:6d87#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #27 qvbc004d01b-cf, fe80::a008:7eff:fe7a:5e47#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #26 qbrc004d01b-cf, fe80::c072:a6ff:fea7:b450#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #25 qbr54d531d7-f3, fe80::3871:9ff:fef7:16f7#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs

edit flag offensive delete link more
0

answered 2012-11-16 01:01:38 -0500

martin-loschwitz gravatar image

What I mean by "disappear" is that they simply are not present in "ip a" anymore, and it looks like openvswitch can't handle them properly:

root@alice:/var/run/openvswitch# grep "No such device" /var/log/openvswitch/ovs-vswitchd.log | wc -l 3420

A lot of entries like this one:

Nov 16 01:57:27|01154|netdev|WARN|failed to get flags for network device tap1935fdda-34: No such device

And numerous other device names.

Are you on Freenode by any chance? My nickname there is madkiss, so if you prefer to communicate that way, we could do that and I will post a summary in here if we manage to solve the problem.

edit flag offensive delete link more
0

answered 2012-11-16 00:51:52 -0500

danwent gravatar image

by "disappear", do you mean they are moved from the root namespace to a different namespace? If so, that is expected. Which actually devices in your example above disappear?

edit flag offensive delete link more
0

answered 2012-11-16 00:42:37 -0500

martin-loschwitz gravatar image

Dan, that is correct. posting before and after is somewhat difficult as the devices disappear the same second they are created.

edit flag offensive delete link more
0

answered 2012-11-16 00:14:48 -0500

martin-loschwitz gravatar image

kvm Am 16.11.2012 01:11 schrieb "yong sheng gong" < question214317@answers.launchpad.net >:

Your question #214317 on quantum changed: https://answers.launchpad.net/quantum/+question/214317 (https://answers.launchpad.net/quantum...)

yong sheng gong requested more information: What hypervisor are u using? Xen?


To answer this request for more information, you can either reply to this email or enter your reply at the following page: https://answers.launchpad.net/quantum/+question/214317 (https://answers.launchpad.net/quantum...)

You received this question notification because you asked the question.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-11-15 22:06:12 -0500

Seen: 65 times

Last updated: Nov 23 '12