NovaException: Unexpected vif_type=binding_failed after upgrading centos 6.5->6.6
Hi there! After upgrading centos 6.5 to 6.6 I cannot create new instance. I get the following error in scheduler.log:
2015-03-03 17:20:53.056 1718 ERROR nova.scheduler.filter_scheduler [req-ea8b64ec-0e73-4fe3-90b0-bfbc520e246f 66a609d76a5d487196f043b06874f910 9f233a10d2ee418b8d7c65b104cf376f] [instance: 619d1698-7bbf-4f98-8d85-ca1c0293e223] Error from last host: compute0 (node compute0): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1328, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 393, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1740, in _spawn\n LOG.exception(_(\'I
nstance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1737, in _spawn\n block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2291, in spawn\n write_to_disk=True)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3480, in to_xml\n disk_info, rescue, block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3294, in get_guest_config\n flavor)\n', u' File "/usr/li
b/python2.6/site-packages/nova/virt/libvirt/vif.py", line 384, in get_config\n _("Unexpected vif_type=%s") % vif_type)\n', u'NovaException: Unexpected vif_type=binding_failed\n']
... which says completely nothing. On compute0 node:
2015-03-03 17:20:48.143 2481 INFO neutron.agent.securitygroups_rpc [req-efcb1c0d-1d2f-4095-b944-3c0246cb97e4 None] Security group member updated [u'ce582782-b451-4aef-b4c6-fde44c33e185']
2015-03-03 17:20:52.803 2481 INFO neutron.agent.securitygroups_rpc [req-b8d0b8de-5ac6-49c3-9217-e9881fda83c7 None] Security group member updated [u'ce582782-b451-4aef-b4c6-fde44c33e185']
2015-03-03 17:20:53.878 2481 INFO neutron.agent.securitygroups_rpc [req-d6ad4800-cec8-4758-8232-fdcfcaf7e020 None] Security group member updated [u'ce582782-b451-4aef-b4c6-fde44c33e185']
2015-03-03 17:20:59.884 2481 INFO neutron.agent.securitygroups_rpc [req-a3c077cb-e4b6-469f-a8d6-d6c6119a270b None] Security group member updated [u'ce582782-b451-4aef-b4c6-fde44c33e185']
2015-03-03 17:21:00.832 2481 INFO neutron.agent.securitygroups_rpc [req-9d492160-f7c0-44ef-a1ee-bff5c7e89cee None] Security group member updated [u'ce582782-b451-4aef-b4c6-fde44c33e185']
2015-03-03 17:21:05.732 2481 INFO neutron.agent.securitygroups_rpc [req-3cd643a5-b8f7-4b52-80d6-52e4699ae59b None] Security group member updated [u'ce582782-b451-4aef-b4c6-fde44c33e185']
This is really strange. I can run my existing instances however I cannot connect them and from console I cannot ping between them (IP address is fine). I was trying to reinstall openvswitch and recreating br-int but it is not giving me any results :( Any help will be appreciated.
UPDATE:
+--------------------------------------+--------------------+----------+-------+----------------+
| id | agent_type | host | alive | admin_state_up |
+--------------------------------------+--------------------+----------+-------+----------------+
| 4ba4397c-0db4-4b80-9b48-0830fed51814 | Open vSwitch agent | compute3 | :-) | True |
| 7565fffa-acbd-4f9a-8d94-51f7eb4684a6 | Open vSwitch agent | compute1 | :-) | True |
| 7e6aadff-f3f8-4ea4-99c1-2facefc54c3c | DHCP agent | network | :-) | True |
| 889fa2f1-fe04-404b-8e64-a9465e809370 | Open vSwitch agent | compute2 | :-) | True |
| 8f9c5875-047a-44cb-800f-9f2fb1633fb9 | L3 agent | network | :-) | True |
| b492e87f-7874-4608-b5f0-42d1ec51a1a2 | Open vSwitch agent | network | :-) | True |
| bebb3a75-0ed6-4493-9766-873e6688cef3 | Metadata agent | network | :-) | True |
| bffa1e6d-7d07-45ab-9d2b-3da4a3797920 | Open vSwitch agent | compute0 | :-) | True |
+--------------------------------------+--------------------+----------+-------+----------------+
whats the output of
Make sure all agents are up and running.
no - all services are fine :( see my update - it is not this...