Instance failing to spawn, logs show `NovaException: Unexpected vif_type=binding_failed`
Afternoon Openstack,
I have hit a problem that has been plaguing me for the past week and a bit, the problem is bellow
2014-09-24 13:35:48.870 2059 ERROR nova.scheduler.filter_scheduler [req-fe9e85a9-b5d2-4977-8f73-dd9d7be3911f 355b18b008fe4fe0a8ab88131cc2b41c 42dac9f04b374fecb0dc48ddba89a2f3] [instance: d9b3c4d7-5331-490c-9cae-9084f391fda2] Error from last host: vmhost3 (node vmhost3.bluereef.com.au): [u'Traceback (most recent call last):\n',
u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1329, in _build_instance\n set_access_ip=set_access_ip)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function\n return function(self, context, *args, **kwargs)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1741, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1738, in _spawn\n block_device_info)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2284, in spawn\n write_to_disk=True)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3478, in to_xml\n disk_info, rescue, block_device_info)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3292, in get_guest_config\n flavor)\n',
u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 384, in get_config\n _("Unexpected vif_type=%s") % vif_type)\n', u'NovaException: Unexpected vif_type=binding_failed\n']
Since that is really a wall of text, here is a TL;DR of the error NovaException: Unexpected vif_type=binding_failed.
Yes, all agents are running:
+----------...+--------------------+---------+-------+----------------+
| id ...| agent_type | host | alive | admin_state_up |
+----------...+--------------------+---------+-------+----------------+
| 0b370b66-...| Open vSwitch agent | os-net | :-) | True |
| 2011c9d5-...| L3 agent | os-net | :-) | True |
| 351cd9d8-...| Open vSwitch agent | vmhost3 | :-) | True |
| 861258be-...| Metadata agent | os-net | :-) | True |
| b71622ce-...| DHCP agent | os-net | :-) | True |
+----------...+--------------------+---------+-------+----------------+
And, yes, all the ML2 and neutron configs are setup correctly, but just in case.
Control Node: http://pastebin.com/wRYTseW2
Network Node: http://pastebin.com/wvg1wpZJ
Compute Node: http://pastebin.com/YyyPng97
If there is anything else I need to put to help debug I can.
Any and all help is welcome and thanked!
The
vif_type=binding failedmessage often indicates a problem with your Neutron configuration. Examine the neutron logs on your compute hosts and on your controller(s) to see if there are any errors that correspond to the timestamps in the compute log.Just spun up a VM while watching the neutron logs on both servers, no errors were thrown in them. But if I do the same thing but watch the nova logs instead the error shows there. I really have no clue why this is happening o_O
Just to be sure...by "both servers", do you mean "all three servers"? If you restart neutron services, are there any errors/tracebacks/etc at the time the service starts?
Ok, I stopped all neutron services on all servers, cleaned out the logs, started the services again and tailed all the logs, I ended up finding this on the compute node. http://pastebin.com/bHcie0zV
It looks like openvswitch has not been started.