Ask Your Question
1

Unable to delete instances in 'ERROR' state

asked 2014-01-29 13:49:40 -0500

jzwiep gravatar image

Every so often we'll be unable to delete 'ERROR'd instances on our OpenStack havana deployment. When trying to delete the instance in question, the following appears in /var/log/nova/nova-compute.log on the instance's host:

    2014-01-29 11:41:00.970 11857 AUDIT nova.compute.manager [req-36e7198e-006e-4462-9f02-d8dc8f88389f 650e1218f88341e882c24bb98aba5425 c9de2ba323d8459aae4106cb048f2743] [instance: 8c6c0227-1b55-4075-b2b5-3666bb7d4164] Terminating instance
    2014-01-29 11:41:01.204 11857 ERROR nova.virt.libvirt.driver [-] [instance: 8c6c0227-1b55-4075-b2b5-3666bb7d4164] During wait destroy, instance disappeared.
    2014-01-29 11:41:01.207 11857 INFO nova.virt.libvirt.firewall [req-36e7198e-006e-4462-9f02-d8dc8f88389f 650e1218f88341e882c24bb98aba5425 c9de2ba323d8459aae4106cb048f2743] [instance: 8c6c0227-1b55-4075-b2b5-3666bb7d4164] Attempted to unfilter instance which is not filtered
    2014-01-29 11:41:01.400 11857 INFO nova.virt.libvirt.driver [req-36e7198e-006e-4462-9f02-d8dc8f88389f 650e1218f88341e882c24bb98aba5425 c9de2ba323d8459aae4106cb048f2743] [instance: 8c6c0227-1b55-4075-b2b5-3666bb7d4164] Deletion of /var/lib/nova/instances/8c6c0227-1b55-4075-b2b5-3666bb7d4164 complete

Then after a short wait, we see an RPC timeout appear in the logs:

2014-01-29 11:42:01.511 11857 ERROR nova.compute.manager [req-36e7198e-006e-4462-9f02-d8dc8f88389f 650e1218f88341e882c24bb98aba5425 c9de2ba323d8459aae4106cb048f2743] [instance: 8c6c0227-1b55-4075-b2b5-3666bb7d4164] Failed to deallocate network for instance.
2014-01-29 11:42:02.170 11857 ERROR nova.openstack.common.rpc.amqp [req-36e7198e-006e-4462-9f02-d8dc8f88389f 650e1218f88341e882c24bb98aba5425 c9de2ba323d8459aae4106cb048f2743] Exception during message handling
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     **args)
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     payload)
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     pass
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-01-29 11:42:02.170 11857 TRACE nova.openstack.common.rpc.amqp ...
(more)
edit retag flag offensive close merge delete

Comments

yes resetting might help - in fact it worked with me. BUT only when resetting into an active state: nova reset-state --active instancename

Brenne gravatar imageBrenne ( 2014-01-30 05:42:17 -0500 )edit

The super bad hack would be cleaning the DB entry manually. :)

koolhead17 gravatar imagekoolhead17 ( 2014-01-30 12:09:04 -0500 )edit

Did you resolve this ? I am seeing exactly the same error in my logs with kilo, on a 3 node install. The instances get into an error state on creation, and never boot, then when I try and delete the logs and timeouts appear identical to the problems you have above. I'm using Neutron ....

rcs123 gravatar imagercs123 ( 2015-08-11 03:27:47 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-01-29 21:11:50 -0500

dheeru gravatar image

Try resetting the instance and then delete. It may help.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-01-29 13:49:40 -0500

Seen: 7,709 times

Last updated: Jan 29 '14