Ask Your Question
1

Linuxbridge agent failing on VM shutdown

asked 2020-07-09 07:56:49 -0500

firelumet gravatar image

After shutting down a virtual machine, either through forced power off from Nova or a "native shutdown", the linuxbridge agent in the compute node gets stuck in a loop attempting to remove the virtual network card:

2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-4807e2e5-4154-494e-816c-c93e4b5254b4 - - - - -] Error in agent loop. Devices info: {'current': set(), 'timestamps': {}, 'added': set(), 'removed': {'tapf0de54fe-19'}, 'updated': set()}: neutron_lib.exceptions.ProcessExecutionError: Exit code: 4; Stdin: ; Stdout: ; Stderr: ebtables v1.8.4 (nf_tables):  CHAIN_USER_DEL failed (Device or resource busy): chain neutronARP-tapf0de54fe-19
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     sync = self.process_network_devices(device_info)
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 217, in process_network_devices
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     resync_b = self.treat_devices_removed(device_info['removed'])
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 368, in treat_devices_removed
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.mgr.delete_arp_spoofing_protection(devices)
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 868, in delete_arp_spoofing_protection
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     arp_protect.delete_arp_spoofing_protection(devices)
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return f(*args, **kwargs)
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py", line 74, in delete_arp_spoofing_protection
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent     chain='PREROUTING')
2020-07-09 08:31:26.505 1352 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py", line 91, in _delete_arp_spoofing_protection ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
1

answered 2020-07-11 17:54:55 -0500

lu1as gravatar image

Hi @firelumet,

I have the same problem since CentOS 8 and ebtables. Seems like the chains need to be flushed before deleting, as you already mentioned. I'm not a contributor, but I looked at the code and adding ebtables(['-F', chain_name(vif)], table=table) before ebtables(['-X', chain_name(vif)], table=table) (line 91) and ebtables(['-F', chain], table=table) before ebtables(['-X', chain], table=table) (line 218) in neuton/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py works for me. In CentOS the full path of the python script is /usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py. I'll try to rise a merge request also, but hope that helps for the meanwhile.

edit flag offensive delete link more

Comments

I reported the https://bugs.launchpad.net/neutron/+bug/1887281 (bug) and created a https://review.opendev.org/#/c/740588/ (change)

lu1as gravatar imagelu1as ( 2020-07-12 07:24:24 -0500 )edit

Thank you for your answer and it did fix it for us.

firelumet gravatar imagefirelumet ( 2020-07-13 10:14:38 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

Stats

Asked: 2020-07-09 07:56:49 -0500

Seen: 72 times

Last updated: Jul 11