Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Error when starting neutron-l3-agent and neutron-dhcp-agent in LXD container

Currently I am running LXD on Ubuntu 15.10. I have created two containers running 16.04, controller and compute. Ive gone ahead and installed and setup all services. The only issue I am having is with neutron-l3-agent and neutron-dhcp-agent. After completing configuration I am seeing the following errors:

neutron-l3-agent.log:

2016-05-09 19:31:40.558 5045 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'add', 'tap2d9c4091-9c', 'type', 'veth', 'peer', 'name', 'qr-2d9c4091-9c', 'netns', 'qrouter-9b021858-d363-4e0b-9691-6cedc2af5bcb'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:84
2016-05-09 19:31:41.390 5045 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists

2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 371, in call
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 960, in process
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 477, in _process_internal_ports
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info self.internal_network_added(p)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 400, in internal_network_added
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info mtu=port.get('mtu'))
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 374, in _internal_network_added
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info prefix=prefix, mtu=mtu)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 248, in plug
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info bridge, namespace, prefix, mtu)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 447, in plug_new
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info namespace2=namespace)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 174, in add_veth
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info self._as_root([], 'link', tuple(args))
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info
2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '9b021858-d363-4e0b-9691-6cedc2af5bcb'
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in _process_router_update
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 436, in _process_router_if_compatible
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 450, in _process_updated_router
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent ri.process(self)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 374, in call
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self.logger(e)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self.force_reraise()
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 371, in call
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 960, in process
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 477, in _process_internal_ports
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self.internal_network_added(p)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 400, in internal_network_added
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent mtu=port.get('mtu'))
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 374, in _internal_network_added
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent prefix=prefix, mtu=mtu)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 248, in plug
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent bridge, namespace, prefix, mtu)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 447, in plug_new
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent namespace2=namespace)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 174, in add_veth
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent self._as_root([], 'link', tuple(args))
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent raise RuntimeError(msg)
2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists

From neutron-dhcp-agent.log:

2016-05-09 19:35:37.743 14807 INFO neutron.agent.dhcp.agent [-] Finished network 04368988-45c0-4e85-af4c-8348d2533a0b dhcp configuration
2016-05-09 19:35:38.212 14807 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 1c75e4aaaef14db28ebc733074f2ad06 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 04735962-e569-470a-8247-5a316be5839b.
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 208, in enable
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1240, in setup
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent self.plugin.release_dhcp_port(network.id, port.device_id)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent self.force_reraise()
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent six.reraise(self.type_, self.value, self.tb)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1234, in setup
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent mtu=network.get('mtu'))
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 248, in plug
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent bridge, namespace, prefix, mtu)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 449, in plug_new
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent ns_veth.link.set_address(mac_address)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 499, in set_address
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent self._as_root([], ('set', self.name, 'address', mac_address))
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 365, in _as_root
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent use_root_namespace=use_root_namespace)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent log_fail_as_error=self.log_fail_as_error)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent log_fail_as_error=log_fail_as_error)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent raise RuntimeError(msg)
2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: mount of /sys failed: Operation not permitted

output of ip netns:

root@controller:/sys# ip netns
qdhcp-04368988-45c0-4e85-af4c-8348d2533a0b (id: 3)
qdhcp-04735962-e569-470a-8247-5a316be5839b (id: 2)
qrouter-9b021858-d363-4e0b-9691-6cedc2af5bcb (id: 1)

l3 agent config:

root@controller:~# cat /etc/neutron/l3_agent.ini |grep -v "#"|grep -v -e '^$'
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
external_network_bridge =
debug = true
verbose = true
[AGENT]

ml2_conf:

root@controller:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -v "#"|grep -v -e '^$'
[DEFAULT]
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = linuxbridge,l2population
extension_drivers = port_security
[ml2_type_flat]
flat_networks = provider
[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]
[ml2_type_vxlan]
vni_ranges = 1:1000
[securitygroup]
enable_ipset = true

linux_bridge

root@controller:~# cat /etc/neutron/plugins/ml2/linuxbridge_agent.ini |grep -v "#"|grep -v -e '^$'
[DEFAULT]
[agent]
[linux_bridge]
physical_interface_mappings = provider:eth0
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
enable_security_group = true
[vxlan]
enable_vxlan = true
local_ip = 172.16.1.51
l2_population = true

root@controller:~# dpkg -l|grep neutron
ii neutron-common 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-linuxbridge-agent 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - linuxbridge agent
ii neutron-metadata-agent 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.0.0-0ubuntu1 all Neutron is a virtual network service for Openstack - server

uname -a output:

Linux controller 4.2.0-35-generic #40-Ubuntu SMP Tue Mar 15 22:15:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Output of neutron agent-list:

root@controller:~# neutron agent-list
+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
| 0306e927-47c2-4d23-830b-94dc3e0d3ded | L3 agent | controller | nova | :-) | True | neutron-l3-agent |
| 32d006e1-64ac-4d5f-a489-3f727d543509 | Linux bridge agent | controller | | :-) | True | neutron-linuxbridge-agent |
| 77ff389b-69a4-4d5f-8fb4-0fa48a6ecca2 | DHCP agent | controller | nova | :-) | True | neutron-dhcp-agent |
| ec4dd465-9e1c-47e5-a5c9-9c795f8c5b35 | Linux bridge agent | compute | | :-) | True | neutron-linuxbridge-agent |
| f397f3a2-22d4-405c-a8bb-715e7e093f04 | Metadata agent | controller | | :-) | True | neutron-metadata-agent |
+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
root@controller:~#