Deleted VMs still showing in nova - Dashboard usage now out-of-sync

asked 2014-02-13 17:35:55 -0600

scarecrow gravatar image

For some reason after fixing some other issues on two KVM nodes, when I create a new instance and delete it using the "nova delete <uuid>" command, the VM is removed on the KVM node ("virsh list" comes up empty), but the controller node still shows it active when issuing the "nova list" command.

I've been able to go directly to the database and set all of the state params (vm_state, task_state, and deleted) to "deleted", NULL, and 0, respectively etc from the instances table. That successfully stopped them from showing up when running "nova list".

However, after doing that a few times, now horizon shows that all of my instance quota is used up and I cannot spawn any more VMs.

So: 1) How do I fix horizon so it accurately shows my running VMs? How can I get it to re-read the instances tables and update the "quota_usages" tables or am I going to have to manually update that.

2) Any ideas on why the compute node is successfully stopping the VM but the controller node is not getting updated or updating its DB.

The only log I have to go by at the moment on is this entry on the nova-compute node:

nova-compute.log.1:2014-02-12 15:21:14.306 47313 ERROR [req-c361a981-2a0e-4e70-b581-789169f87455 81fa9eb39aad4ec4bab77cd4c44664de 5a4e09f14dec4361870c1756edb28a45] [instance: 21c6fb58-c7d2-4602-bf8e-c7c282c9b3ec] Failed storing info cache
nova-compute.log.1:2014-02-12 15:21:16.052 47313 ERROR nova.virt.driver [-] Exception dispatching event <nova.virt.event.LifecycleEvent object at 0x3e71d10>: Version 1.9 of Instance is not supported
nova-compute.log.1:2014-02-12 15:21:16.728 47313 ERROR nova.openstack.common.rpc.amqp [req-c361a981-2a0e-4e70-b581-789169f87455 81fa9eb39aad4ec4bab77cd4c44664de 5a4e09f14dec4361870c1756edb28a45] Exception during message handling

Any ideas would be appreciated.

edit retag flag offensive close merge delete


Seems to be bug atleast for me. You can file bug one does not exist with exact signature.

dheeru gravatar imagedheeru ( 2014-02-13 21:03:51 -0600 )edit

I am running Kilo and have this same damned problem. How hard is it for the 'delete instance' process to decrement the "instances in use" datum?

tiger gravatar imagetiger ( 2017-04-27 17:38:32 -0600 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2014-02-14 11:17:06 -0600

gabriel_staicu gravatar image

I encountered the same problem and I found a solution on but now I cannot find the answer/question to give credit. This was the solution: In the database nova, table quota_usage for ram, cpu, etc put the value -1 and the quota will update in horizon and you will be able to create new instances.

edit flag offensive delete link more


Thanks for the pointer. I had completely missed this post: Ok, so now that that's fixed, does anyon have any insight on Question 2)? I can re-post as another question if that is more appropriate.

scarecrow gravatar imagescarecrow ( 2014-02-14 14:13:45 -0600 )edit

Anybody have any ideas on why virsh is successfully terminating the VM but the controller is not? The controller goes into a brief state of "deleting" for the instances task state , but the VM always stays ACTIVE and the only way to kill it is to hack the database?

scarecrow gravatar imagescarecrow ( 2014-02-17 17:27:33 -0600 )edit

And I'll just add that I'm seeing this on 2 of the 3 KVM nodes that we have.

scarecrow gravatar imagescarecrow ( 2014-02-17 17:28:48 -0600 )edit

Does anyone know if there's an existing bug for this? If not I'd like to write one up.

Daniel P gravatar imageDaniel P ( 2014-07-30 09:17:49 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-02-13 17:35:55 -0600

Seen: 2,862 times

Last updated: Feb 13 '14