Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

DVR: Failed updating arp entry

Using neutron DVR in Kilo, new router namespace is unable to ping DHCP servers. Problem occurs on one host only. All other hosts with the same network can reach the dhcp servers. And all other networks on the same host can reach their DHCP servers. I see the following error in the l3-agent log file.

2015-07-20 08:56:17.302 94525 ERROR neutron.agent.linux.utils [req-36b3da49-9d7f-4c83-90c0-b7e969493636 ]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5f1d4930-46d4-42f6-a580-3eefd4e24bf8', 'ip', '-4', 'neigh', 'replace', '172.16.0.8', 'lladdr', 'fa:16:3e:3b:67:69', 'nud', 'permanent', 'dev', 'qr-cef89d3d-68']
Exit code: 1
Stdin:
Stdout:
Stderr: Cannot find device "qr-cef89d3d-68"

2015-07-20 08:56:17.303 94525 ERROR neutron.agent.l3.dvr_router [req-36b3da49-9d7f-4c83-90c0-b7e969493636 ] DVR: Failed updating arp entry
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router Traceback (most recent call last):
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router.py", line 195, in _update_arp_entry
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router     device.neigh.add(ip, mac)
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 538, in add
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router     'dev', self.name))
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router     use_root_namespace=use_root_namespace)
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=self.log_fail_as_error)
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=log_fail_as_error)
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router     raise RuntimeError(m)
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router RuntimeError:
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5f1d4930-46d4-42f6-a580-3eefd4e24bf8', 'ip', '-4', 'neigh', 'replace', '172.16.0.8', 'lladdr', 'fa:16:3e:3b:67:69', 'nud', 'permanent', 'dev', 'qr-cef89d3d-68']
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router Exit code: 1
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router Stdin:
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router Stdout:
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router Stderr: Cannot find device "qr-cef89d3d-68"
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router
2015-07-20 08:56:17.303 94525 TRACE neutron.agent.l3.dvr_router
2015-07-20 08:56:17.304 94525 ERROR oslo_messaging.rpc.dispatcher [req-36b3da49-9d7f-4c83-90c0-b7e969493636 ] Exception during message handling:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5f1d4930-46d4-42f6-a580-3eefd4e24bf8', 'ip', '-4', 'neigh', 'replace', '172.16.0.8', 'lladdr', 'fa:16:3e:3b:67:69', 'nud', 'permanent', 'dev', 'qr-cef89d3d-68']
Exit code: 1
Stdin:
Stdout:
Stderr: Cannot find device "qr-cef89d3d-68"
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr.py", line 71, in add_arp_entry
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     ri._update_arp_entry(ip, mac, subnet_id, 'add')
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router.py", line 200, in _update_arp_entry
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     LOG.exception(_LE("DVR: Failed updating arp entry"))
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router.py", line 195, in _update_arp_entry
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     device.neigh.add(ip, mac)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 538, in add
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     'dev', self.name))
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     use_root_namespace=use_root_namespace)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     log_fail_as_error=self.log_fail_as_error)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     log_fail_as_error=log_fail_as_error)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher     raise RuntimeError(m)
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher RuntimeError:
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5f1d4930-46d4-42f6-a580-3eefd4e24bf8', 'ip', '-4', 'neigh', 'replace', '172.16.0.8', 'lladdr', 'fa:16:3e:3b:67:69', 'nud', 'permanent', 'dev', 'qr-cef89d3d-68']
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher Exit code: 1
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher Stdin:
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher Stdout:
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher Stderr: Cannot find device "qr-cef89d3d-68"
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher
2015-07-20 08:56:17.304 94525 TRACE oslo_messaging.rpc.dispatcher