Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

VM Failed to evacuate from crash compute node.

hi guys,

i have a problem to evacuate one of the vm to the active compute node. basically i just test to see if it is possible to evacuate an instance/vm to an active compute node if the existing compute node stop functioning. all my vm are stored inside ceph storage. i have 2 compute node and 1 management. all 3 servers are actively running ceph as a cluster. when i simulate one of my compute node down. and i run nova evacuate i got below error.

 [req-de5ea93c-a9e2-44cb-9ba9-2da018181013 ] Exception during message handling: Virtual Interface creation failed
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     payload)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     pass
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2831, in rebuild_instance
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     self._rebuild_default_impl(**kwargs)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2685, in _rebuild_default_impl
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     block_device_info=new_block_device_info)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2621, in spawn
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     block_device_info, disk_info=disk_info)
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4435, in _create_domain_and_network
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher     raise exception.VirtualInterfaceCreateException()
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher VirtualInterfaceCreateException: Virtual Interface creation failed
2015-01-16 17:06:58.310 6560 TRACE oslo.messaging.rpc.dispatcher
2015-01-16 17:07:02.179 6560 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:193
2015-01-16 17:07:02.179 6560 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-01-16 17:07:02.180 6560 DEBUG nova.virt.libvirt.driver [-] Updating host stats update_status /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:6388

any suggestion how to troubleshoot this? please advise.