Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Boot instance failed with error 'Instance failed network setup'

I deployed an openstack by using one server as both controller/network node and another as compute node, follows openstack install-guide. All verification passed until I finished neutron installation/configuration and tried to launch an instance for testing.

It always failed with error on network setup, instance stuck in error status and can't be deleted, content in /var/log/nova/nova-compute.log attached below.

2015-03-24 10:47:31.218 32574 AUDIT nova.compute.manager [req-6a0db069-b46a-4e9d-8360-aaff5cd7e160 None] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Starting instance...
2015-03-24 10:47:31.275 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Attempting claim: memory 512 MB, disk 1 GB
2015-03-24 10:47:31.275 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Total memory: 144716 MB, used: 2048.00 MB
2015-03-24 10:47:31.276 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] memory limit: 217074.00 MB, free: 215026.00 MB
2015-03-24 10:47:31.276 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Total disk: 49 GB, used: 3.00 GB
2015-03-24 10:47:31.276 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] disk limit not specified, defaulting to unlimited
2015-03-24 10:47:31.285 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Claim successful
2015-03-24 10:47:31.634 32574 INFO nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Creating image
2015-03-24 10:47:31.694 32574 WARNING nova.virt.disk.vfs.guestfs [-] Unable to force TCG mode, libguestfs too old? 'GuestFS' object has no attribute 'set_backend_settings'
2015-03-24 10:47:34.104 32574 WARNING nova.virt.disk.vfs.guestfs [-] Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas
2015-03-24 10:48:31.448 32574 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager Traceback (most recent call last):
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1646, in _allocate_network_async
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapped
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     return func(self, context, *args, **kwargs)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 61, in wrapper
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     res = f(self, context, *args, **kwargs)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 283, in allocate_for_instance
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 190, in allocate_for_instance
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     macs=jsonutils.to_primitive(macs))
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     retry=self.retry)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     timeout=timeout, retry=retry)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     retry=retry)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     result = self._waiter.wait(msg_id, timeout)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 298, in wait
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     reply, ending, trylock = self._poll_queue(msg_id, timeout)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 238, in _poll_queue
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     message = self.waiters.get(msg_id, timeout)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 144, in get
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     'to message ID %s' % msg_id)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID 2594ee96b05f44e2bd0493b9f470d208
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager
2015-03-24 10:48:31.450 32574 ERROR nova.compute.manager [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Instance failed to spawn
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Traceback (most recent call last):
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2246, in _build_resources
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     yield resources
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2116, in _build_and_run_instance
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     block_device_info=block_device_info)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2620, in spawn
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     write_to_disk=True)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4145, in _get_guest_xml
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     network_info_str = str(network_info)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 467, in __str__
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return self._sync_wrapper(fn, *args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 450, in _sync_wrapper
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     self.wait()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 482, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     self[:] = self._gt.wait()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 173, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return self._exit_event.wait()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return hubs.get_hub().switch()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 293, in switch
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return self.greenlet.switch()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     result = function(*args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1646, in _allocate_network_async
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     dhcp_options=dhcp_options)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapped
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return func(self, context, *args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 61, in wrapper
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     res = f(self, context, *args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 283, in allocate_for_instance
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 190, in allocate_for_instance
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     macs=jsonutils.to_primitive(macs))
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     retry=self.retry)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     timeout=timeout, retry=retry)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     retry=retry)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     result = self._waiter.wait(msg_id, timeout)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 298, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     reply, ending, trylock = self._poll_queue(msg_id, timeout)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 238, in _poll_queue
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     message = self.waiters.get(msg_id, timeout)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 144, in get
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     'to message ID %s' % msg_id)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] MessagingTimeout: Timed out waiting for a reply to message ID 2594ee96b05f44e2bd0493b9f470d208
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]
2015-03-24 10:48:31.451 32574 AUDIT nova.compute.manager [req-6a0db069-b46a-4e9d-8360-aaff5cd7e160 None] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Terminating instance
2015-03-24 10:48:31.457 32574 WARNING nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] During wait destroy, instance disappeared.
2015-03-24 10:48:31.457 32574 INFO nova.virt.libvirt.firewall [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Attempted to unfilter instance which is not filtered
2015-03-24 10:48:31.552 32574 INFO nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Deleting instance files /var/lib/nova/instances/3f84e1ed-eacf-4d09-9059-c84941798090_del
2015-03-24 10:48:31.552 32574 INFO nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Deletion of /var/lib/nova/instances/3f84e1ed-eacf-4d09-9059-c84941798090_del complete

Boot instance failed with error 'Instance failed network setup'

I deployed an openstack by using one server as both controller/network node and another as compute node, follows openstack install-guide. All verification passed until I finished neutron installation/configuration and tried to launch an instance for testing.

It always failed with error on network setup, instance stuck in error status and can't be deleted, content in /var/log/nova/nova-compute.log attached below.

2015-03-24 10:47:31.218 32574 AUDIT nova.compute.manager [req-6a0db069-b46a-4e9d-8360-aaff5cd7e160 None] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Starting instance...
2015-03-24 10:47:31.275 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Attempting claim: memory 512 MB, disk 1 GB
2015-03-24 10:47:31.275 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Total memory: 144716 MB, used: 2048.00 MB
2015-03-24 10:47:31.276 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] memory limit: 217074.00 MB, free: 215026.00 MB
2015-03-24 10:47:31.276 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Total disk: 49 GB, used: 3.00 GB
2015-03-24 10:47:31.276 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] disk limit not specified, defaulting to unlimited
2015-03-24 10:47:31.285 32574 AUDIT nova.compute.claims [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Claim successful
2015-03-24 10:47:31.634 32574 INFO nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Creating image
2015-03-24 10:47:31.694 32574 WARNING nova.virt.disk.vfs.guestfs [-] Unable to force TCG mode, libguestfs too old? 'GuestFS' object has no attribute 'set_backend_settings'
2015-03-24 10:47:34.104 32574 WARNING nova.virt.disk.vfs.guestfs [-] Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas
2015-03-24 10:48:31.448 32574 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager Traceback (most recent call last):
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1646, in _allocate_network_async
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapped
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     return func(self, context, *args, **kwargs)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 61, in wrapper
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     res = f(self, context, *args, **kwargs)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 283, in allocate_for_instance
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 190, in allocate_for_instance
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     macs=jsonutils.to_primitive(macs))
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     retry=self.retry)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     timeout=timeout, retry=retry)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     retry=retry)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     result = self._waiter.wait(msg_id, timeout)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 298, in wait
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     reply, ending, trylock = self._poll_queue(msg_id, timeout)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 238, in _poll_queue
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     message = self.waiters.get(msg_id, timeout)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 144, in get
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager     'to message ID %s' % msg_id)
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID 2594ee96b05f44e2bd0493b9f470d208
2015-03-24 10:48:31.448 32574 TRACE nova.compute.manager
2015-03-24 10:48:31.450 32574 ERROR nova.compute.manager [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Instance failed to spawn
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Traceback (most recent call last):
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2246, in _build_resources
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     yield resources
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2116, in _build_and_run_instance
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     block_device_info=block_device_info)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2620, in spawn
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     write_to_disk=True)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4145, in _get_guest_xml
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     network_info_str = str(network_info)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 467, in __str__
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return self._sync_wrapper(fn, *args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 450, in _sync_wrapper
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     self.wait()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 482, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     self[:] = self._gt.wait()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 173, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return self._exit_event.wait()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return hubs.get_hub().switch()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 293, in switch
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return self.greenlet.switch()
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     result = function(*args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1646, in _allocate_network_async
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     dhcp_options=dhcp_options)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapped
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     return func(self, context, *args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 61, in wrapper
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     res = f(self, context, *args, **kwargs)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 283, in allocate_for_instance
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 190, in allocate_for_instance
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     macs=jsonutils.to_primitive(macs))
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     retry=self.retry)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     timeout=timeout, retry=retry)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     retry=retry)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     result = self._waiter.wait(msg_id, timeout)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 298, in wait
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     reply, ending, trylock = self._poll_queue(msg_id, timeout)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 238, in _poll_queue
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     message = self.waiters.get(msg_id, timeout)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 144, in get
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]     'to message ID %s' % msg_id)
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] MessagingTimeout: Timed out waiting for a reply to message ID 2594ee96b05f44e2bd0493b9f470d208
2015-03-24 10:48:31.450 32574 TRACE nova.compute.manager [instance: 3f84e1ed-eacf-4d09-9059-c84941798090]
2015-03-24 10:48:31.451 32574 AUDIT nova.compute.manager [req-6a0db069-b46a-4e9d-8360-aaff5cd7e160 None] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Terminating instance
2015-03-24 10:48:31.457 32574 WARNING nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] During wait destroy, instance disappeared.
2015-03-24 10:48:31.457 32574 INFO nova.virt.libvirt.firewall [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Attempted to unfilter instance which is not filtered
2015-03-24 10:48:31.552 32574 INFO nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Deleting instance files /var/lib/nova/instances/3f84e1ed-eacf-4d09-9059-c84941798090_del
2015-03-24 10:48:31.552 32574 INFO nova.virt.libvirt.driver [-] [instance: 3f84e1ed-eacf-4d09-9059-c84941798090] Deletion of /var/lib/nova/instances/3f84e1ed-eacf-4d09-9059-c84941798090_del complete

output of virsh nwfilter-list,

    UUID                                  Name
    ----------------------------------------------------------------
    d94cb7f8-9e9a-48bd-9c40-248e68fbdd1d  allow-arp
    41c80833-d40e-4eb0-954c-68d04feab8e1  allow-dhcp
    2a535893-6b1f-4d60-9e4e-5d82e4a783b9  allow-dhcp-server
    e32a93b4-b641-4c87-93f7-aaf3d45c5b49  allow-incoming-ipv4
    33354ba9-fe7a-44e7-ab79-5b045e839406  allow-ipv4
    9e19034e-f038-4b9f-bea9-1c345a702299  clean-traffic
    5e4a47ed-7411-42a2-8dab-7723be4b62df  no-arp-ip-spoofing
    eee4b66a-0cc9-401c-8947-698be71094b2  no-arp-mac-spoofing
    7989b75f-b6a5-43b7-986c-6219aa713f30  no-arp-spoofing
    6819bdee-eaeb-4fbe-ae82-7e4b9e5e257e  no-ip-multicast
    980d39d2-c140-45d0-a802-57e54ca7ff91  no-ip-spoofing
    dfea9c47-7896-4181-a5aa-e0d0426f79bf  no-mac-broadcast
    8c5db1cc-57bc-48bf-a34a-ab97fe9802f5  no-mac-spoofing
    4040208e-8faf-41f4-a9a6-0043036cbb37  no-other-l2-traffic
    63f29ba3-cbd0-4572-a429-eb15f76a5856  no-other-rarp-traffic
    42acdcf0-b048-4c5d-9877-cdd1cdf0dfcd  qemu-announce-self
    3ca0dab3-5e86-46b4-a1eb-32bef59fbf31  qemu-announce-self-rarp

Tables in Nova DB,

MariaDB [nova]> show tables
    -> \g
+--------------------------------------------+
| Tables_in_nova                             |
+--------------------------------------------+
| agent_builds                               |
| aggregate_hosts                            |
| aggregate_metadata                         |
| aggregates                                 |
| block_device_mapping                       |
| bw_usage_cache                             |
| cells                                      |
| certificates                               |
| compute_nodes                              |
| console_pools                              |
| consoles                                   |
| dns_domains                                |
| fixed_ips                                  |
| floating_ips                               |
| instance_actions                           |
| instance_actions_events                    |
| instance_extra                             |
| instance_faults                            |
| instance_group_member                      |
| instance_group_policy                      |
| instance_groups                            |
| instance_id_mappings                       |
| instance_info_caches                       |
| instance_metadata                          |
| instance_system_metadata                   |
| instance_type_extra_specs                  |
| instance_type_projects                     |
| instance_types                             |
| instances                                  |
| iscsi_targets                              |
| key_pairs                                  |
| migrate_version                            |
| migrations                                 |
| networks                                   |
| pci_devices                                |
| project_user_quotas                        |
| provider_fw_rules                          |
| quota_classes                              |
| quota_usages                               |
| quotas                                     |
| reservations                               |
| s3_images                                  |
| security_group_default_rules               |
| security_group_instance_association        |
| security_group_rules                       |
| security_groups                            |
| services                                   |
| shadow_agent_builds                        |
| shadow_aggregate_hosts                     |
| shadow_aggregate_metadata                  |
| shadow_aggregates                          |
| shadow_block_device_mapping                |
| shadow_bw_usage_cache                      |
| shadow_cells                               |
| shadow_certificates                        |
| shadow_compute_nodes                       |
| shadow_console_pools                       |
| shadow_consoles                            |
| shadow_dns_domains                         |
| shadow_fixed_ips                           |
| shadow_floating_ips                        |
| shadow_instance_actions                    |
| shadow_instance_actions_events             |
| shadow_instance_extra                      |
| shadow_instance_faults                     |
| shadow_instance_group_member               |
| shadow_instance_group_policy               |
| shadow_instance_groups                     |
| shadow_instance_id_mappings                |
| shadow_instance_info_caches                |
| shadow_instance_metadata                   |
| shadow_instance_system_metadata            |
| shadow_instance_type_extra_specs           |
| shadow_instance_type_projects              |
| shadow_instance_types                      |
| shadow_instances                           |
| shadow_iscsi_targets                       |
| shadow_key_pairs                           |
| shadow_migrate_version                     |
| shadow_migrations                          |
| shadow_networks                            |
| shadow_pci_devices                         |
| shadow_project_user_quotas                 |
| shadow_provider_fw_rules                   |
| shadow_quota_classes                       |
| shadow_quota_usages                        |
| shadow_quotas                              |
| shadow_reservations                        |
| shadow_s3_images                           |
| shadow_security_group_default_rules        |
| shadow_security_group_instance_association |
| shadow_security_group_rules                |
| shadow_security_groups                     |
| shadow_services                            |
| shadow_snapshot_id_mappings                |
| shadow_snapshots                           |
| shadow_task_log                            |
| shadow_virtual_interfaces                  |
| shadow_volume_id_mappings                  |
| shadow_volume_usage_cache                  |
| shadow_volumes                             |
| snapshot_id_mappings                       |
| snapshots                                  |
| task_log                                   |
| virtual_interfaces                         |
| volume_id_mappings                         |
| volume_usage_cache                         |
| volumes                                    |
+--------------------------------------------+
108 rows in set (0.00 sec)