Deleted VMs still showing in nova - Dashboard usage now out-of-sync
For some reason after fixing some other issues on two KVM nodes, when I create a new instance and delete it using the "nova delete <uuid>" command, the VM is removed on the KVM node ("virsh list" comes up empty), but the controller node still shows it active when issuing the "nova list" command.
I've been able to go directly to the database and set all of the state params (vm_state, task_state, and deleted) to "deleted", NULL, and 0, respectively etc from the instances table. That successfully stopped them from showing up when running "nova list".
However, after doing that a few times, now horizon shows that all of my instance quota is used up and I cannot spawn any more VMs.
So: 1) How do I fix horizon so it accurately shows my running VMs? How can I get it to re-read the instances tables and update the "quota_usages" tables or am I going to have to manually update that.
2) Any ideas on why the compute node is successfully stopping the VM but the controller node is not getting updated or updating its DB.
The only log I have to go by at the moment on is this entry on the nova-compute node:
nova-compute.log.1:2014-02-12 15:21:14.306 47313 ERROR nova.network.api [req-c361a981-2a0e-4e70-b581-789169f87455 81fa9eb39aad4ec4bab77cd4c44664de 5a4e09f14dec4361870c1756edb28a45] [instance: 21c6fb58-c7d2-4602-bf8e-c7c282c9b3ec] Failed storing info cache
nova-compute.log.1:2014-02-12 15:21:16.052 47313 ERROR nova.virt.driver [-] Exception dispatching event <nova.virt.event.LifecycleEvent object at 0x3e71d10>: Version 1.9 of Instance is not supported
nova-compute.log.1:2014-02-12 15:21:16.728 47313 ERROR nova.openstack.common.rpc.amqp [req-c361a981-2a0e-4e70-b581-789169f87455 81fa9eb39aad4ec4bab77cd4c44664de 5a4e09f14dec4361870c1756edb28a45] Exception during message handling
Any ideas would be appreciated.
Seems to be bug atleast for me. You can file bug one does not exist with exact signature.
I am running Kilo and have this same damned problem. How hard is it for the 'delete instance' process to decrement the "instances in use" datum?