Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Openstack network error - RTNETLINK answers: File exists

Hello I have an Openstack newton cluster.. I can successfully launch instance and all.. But now while launching instance it is always in spawning state and I check the /var/log/neutron/neutron-linuxbridge-agent.log and it has error like below

2016-12-08 17:49:36.302 3506 ERROR neutron.agent.linux.utils [req-ec796998-fb3f-4d56-9e8a-c60936c80d33 - - - - -] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists

2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-ec796998-fb3f-4d56-9e8a-c60936c80d33 - - - - -] Error in agent loop. Devices info: {'current': set(['tape0a2b1a4-cd', 'tap687f2b59-cd', 'tap15bf769e-42', 'tap9a71eee4-22']), 'timestamps': {'tape0a2b1a4-cd': 44, 'tap687f2b59-cd': 48, 'tap9a71eee4-22': 42, 'tap15bf769e-42': 47}, 'removed': set([]), 'added': set(['tape0a2b1a4-cd', 'tap687f2b59-cd', 'tap15bf769e-42', 'tap9a71eee4-22']), 'updated': set([])}
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 452, in daemon_loop
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     sync = self.process_network_devices(device_info)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return f(*args, **kwargs)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 209, in process_network_devices
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     resync_a = self.treat_devices_added_updated(devices_added_updated)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return f(*args, **kwargs)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 226, in treat_devices_added_updated
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._process_device_if_exists(device_details)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 253, in _process_device_if_exists
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, device_details['device_owner'])
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 512, in plug_interface
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     tap_name, device_owner)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 453, in add_tap_interface
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return False
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.force_reraise()
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     six.reraise(self.type_, self.value, self.tb)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 445, in add_tap_interface
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     tap_device_name, device_owner)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 477, in _add_tap_interface
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     segmentation_id)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 429, in ensure_physical_in_bridge
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     physical_interface)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 247, in ensure_flat_bridge
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     gateway):
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 388, in ensure_bridge
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.update_interface_ip_details(bridge_name, interface, ips, gateway)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 327, in update_interface_ip_details
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     dst_device.addr.add(cidr=ip['cidr'])
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 577, in add
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._as_root([net.version], tuple(args))
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     use_root_namespace=use_root_namespace)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     log_fail_as_error=self.log_fail_as_error)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     log_fail_as_error=log_fail_as_error)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise RuntimeError(msg)
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent 
2016-12-08 17:49:36.344 3506 ERROR neutron.plugins.ml2.drivers.agent._common_agent

I also have one error in neutron-server.log

2016-12-08 17:54:05.411 5214 DEBUG neutron.db.api [req-2ddf59a5-0d29-410c-9ab6-5fff4720504d - - - - -] Retry wrapper got retriable exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 119, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1733, in update_port_status
    context.session.flush()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2027, in flush
    self._flush(objects)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2145, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2109, in _flush
    flush_context.execute()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_update_statements
    (table.description, len(records), rows))
StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

I dont know why this happened.. because everything was working fine upto today morning.. Any help is really appreciated..