ovs_neutron_agent MessagingTimeout

asked 2017-11-26 07:23:28 -0500

Mario Sommer gravatar image

After an upgrade MItaka->Newton->Ocata some of my compute nodes can't spawn instances any more. In nova-compute.log I see, that the network does not work.

ERROR nova.compute.manager [req-62abe932-057a-4710-beba-7785fdd1fb52 15d72e13a10f46a5a65359f6c4e42b0f e660c9a7cd624e32944ba0b760a82044 - - -] [instance: fa8c602c-217e-4a28-87b2-36d152cac504] Failed to allocate network(s)
ERROR nova.compute.manager [instance: fa8c602c-217e-4a28-87b2-36d152cac504] VirtualInterfaceCreateException: Virtual Interface creation failed

In openvswitch-agent.log I see, that the neutron times out. Anyone has an idea what could be wrong?

ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-f1f898d1-ac40-4868-b1a4-0e7fd3b4d06d - - - - -] Error while processing VIF ports
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2039, in rpc_loop
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1642, in process_network_ports
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 878, in _bind_devices
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 146, in update_device_list
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_id=agent_id, host=host)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 174, in call
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
    2017-11-26 13:27:25.459 18335 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages ...
(more)
edit retag flag offensive close merge delete

Comments

neutron server is runnning with extreme load (91,4%), since the upgrade. I have no idea why. 7250 neutron 20 0 492648 176712 2304 R 91,4 0,6 6:01.33 neutron-server

Mario Sommer gravatar imageMario Sommer ( 2017-11-26 09:19:13 -0500 )edit