Revision history [back]

click to hide/show revision 1
initial version

neutron-linuxbridge-agent Stuck in Loop - RTNETLINK answers: Operation not supported

I've setup a Openstack following the Ubuntu installation guide. It looks like I have a Neutron issue on my compute node. It's stuck at 100% CPU, looping continuously over the following. Judging by the 'RTNETLINK answers: Operation not supported', it's trying to do something to the network interfaces, but can't - something to do with the vxlan. I've tried rebooting the node, and tracing through the code, but I'm not having much luck.

 2016-04-25 00:04:26.613 17806 INFO neutron.common.config [-] Logging enabled!
2016-04-25 00:04:26.614 17806 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 8.0.0
2016-04-25 00:04:26.614 17806 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'eth0'}
2016-04-25 00:04:26.615 17806 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {}
2016-04-25 00:04:26.810 17806 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Operation not supported

2016-04-25 00:04:26.828 17806 CRITICAL neutron [-] RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Operation not supported

2016-04-25 00:04:26.828 17806 ERROR neutron Traceback (most recent call last):
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/bin/neutron-linuxbridge-agent", line 10, in <module>
2016-04-25 00:04:26.828 17806 ERROR neutron     sys.exit(main())
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/linuxbridge_neutron_agent.py", line 21, in main
2016-04-25 00:04:26.828 17806 ERROR neutron     agent_main.main()
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 888, in main
2016-04-25 00:04:26.828 17806 ERROR neutron     manager = LinuxBridgeManager(bridge_mappings, interface_mappings)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 76, in __init__
2016-04-25 00:04:26.828 17806 ERROR neutron     self.check_vxlan_support()
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 618, in check_vxlan_support
2016-04-25 00:04:26.828 17806 ERROR neutron     if self.vxlan_ucast_supported():
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 581, in vxlan_ucast_supported
2016-04-25 00:04:26.828 17806 ERROR neutron     test_iface = self.ensure_vxlan(seg_id)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 286, in ensure_vxlan
2016-04-25 00:04:26.828 17806 ERROR neutron     return None
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-25 00:04:26.828 17806 ERROR neutron     self.force_reraise()
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-25 00:04:26.828 17806 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 276, in ensure_vxlan
2016-04-25 00:04:26.828 17806 ERROR neutron     **args)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 244, in add_vxlan
2016-04-25 00:04:26.828 17806 ERROR neutron     self._as_root([], 'link', cmd)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-04-25 00:04:26.828 17806 ERROR neutron     log_fail_as_error=self.log_fail_as_error)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-04-25 00:04:26.828 17806 ERROR neutron     log_fail_as_error=log_fail_as_error)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-04-25 00:04:26.828 17806 ERROR neutron     raise RuntimeError(msg)
2016-04-25 00:04:26.828 17806 ERROR neutron RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Operation not supported
2016-04-25 00:04:26.828 17806 ERROR neutron
2016-04-25 00:04:26.828 17806 ERROR neutron

My networking config as it stands:

eth0      Link encap:Ethernet  HWaddr 0c:c4:7a:c5:a7:ba
          inet6 addr: fe80::ec4:7aff:fec5:a7ba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2981 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:178959 (178.9 KB)  TX bytes:438 (438.0 B)

eth1      Link encap:Ethernet  HWaddr 0c:c4:7a:c5:a7:bb
          inet addr:172.16.0.21  Bcast:172.31.255.255  Mask:255.240.0.0
          inet6 addr: fe80::ec4:7aff:fec5:a7bb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:595 errors:0 dropped:0 overruns:0 frame:0
          TX packets:390 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:53687 (53.6 KB)  TX bytes:114639 (114.6 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:98 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7276 (7.2 KB)  TX bytes:7276 (7.2 KB)

virbr0    Link encap:Ethernet  HWaddr 52:54:00:63:d2:c8
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

neutron-linuxbridge-agent Stuck in Loop - RTNETLINK answers: Operation not supported

I've setup a Openstack following the Ubuntu installation guide. It looks like I have a Neutron issue on my compute node. It's stuck at 100% CPU, looping continuously over the following. Judging by the 'RTNETLINK answers: Operation not supported', it's trying to do something to the network interfaces, but can't - something to do with the vxlan. I've tried rebooting the node, and tracing through the code, but I'm not having much luck.

 2016-04-25 00:04:26.613 17806 INFO neutron.common.config [-] Logging enabled!
2016-04-25 00:04:26.614 17806 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 8.0.0
2016-04-25 00:04:26.614 17806 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'eth0'}
2016-04-25 00:04:26.615 17806 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {}
2016-04-25 00:04:26.810 17806 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Operation not supported

2016-04-25 00:04:26.828 17806 CRITICAL neutron [-] RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Operation not supported

2016-04-25 00:04:26.828 17806 ERROR neutron Traceback (most recent call last):
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/bin/neutron-linuxbridge-agent", line 10, in <module>
2016-04-25 00:04:26.828 17806 ERROR neutron     sys.exit(main())
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/linuxbridge_neutron_agent.py", line 21, in main
2016-04-25 00:04:26.828 17806 ERROR neutron     agent_main.main()
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 888, in main
2016-04-25 00:04:26.828 17806 ERROR neutron     manager = LinuxBridgeManager(bridge_mappings, interface_mappings)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 76, in __init__
2016-04-25 00:04:26.828 17806 ERROR neutron     self.check_vxlan_support()
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 618, in check_vxlan_support
2016-04-25 00:04:26.828 17806 ERROR neutron     if self.vxlan_ucast_supported():
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 581, in vxlan_ucast_supported
2016-04-25 00:04:26.828 17806 ERROR neutron     test_iface = self.ensure_vxlan(seg_id)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 286, in ensure_vxlan
2016-04-25 00:04:26.828 17806 ERROR neutron     return None
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-25 00:04:26.828 17806 ERROR neutron     self.force_reraise()
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-25 00:04:26.828 17806 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 276, in ensure_vxlan
2016-04-25 00:04:26.828 17806 ERROR neutron     **args)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 244, in add_vxlan
2016-04-25 00:04:26.828 17806 ERROR neutron     self._as_root([], 'link', cmd)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-04-25 00:04:26.828 17806 ERROR neutron     log_fail_as_error=self.log_fail_as_error)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-04-25 00:04:26.828 17806 ERROR neutron     log_fail_as_error=log_fail_as_error)
2016-04-25 00:04:26.828 17806 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-04-25 00:04:26.828 17806 ERROR neutron     raise RuntimeError(msg)
2016-04-25 00:04:26.828 17806 ERROR neutron RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Operation not supported
2016-04-25 00:04:26.828 17806 ERROR neutron
2016-04-25 00:04:26.828 17806 ERROR neutron

My networking config as it stands:

eth0      Link encap:Ethernet  HWaddr 0c:c4:7a:c5:a7:ba
          inet6 addr: fe80::ec4:7aff:fec5:a7ba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2981 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:178959 (178.9 KB)  TX bytes:438 (438.0 B)

eth1      Link encap:Ethernet  HWaddr 0c:c4:7a:c5:a7:bb
          inet addr:172.16.0.21  Bcast:172.31.255.255  Mask:255.240.0.0
          inet6 addr: fe80::ec4:7aff:fec5:a7bb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:595 errors:0 dropped:0 overruns:0 frame:0
          TX packets:390 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:53687 (53.6 KB)  TX bytes:114639 (114.6 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:98 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7276 (7.2 KB)  TX bytes:7276 (7.2 KB)

virbr0    Link encap:Ethernet  HWaddr 52:54:00:63:d2:c8
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)