nova scheduler scheduling instance on invalid host

asked 2014-10-26 18:39:12 -0600

bdastur-s gravatar image

updated 2014-10-27 02:05:49 -0600

I have an openstack setup with 1 controller and 2 computes. One of the compute nodes is "out-of-service", meaning there is not nova-compute running on the host. The host is still up and accessible.

Now, I have an automated script which exercises the stack by trying to create instances on the hosts. It uses the python nova client APIs to get the available list of of hosts, which still returns the bad compute, and when trying to create an instance on the compute, it hangs.

I tried following the nova-scheduler and nova-api code paths, and I see that in the scheduler, the filter_properties are set with forced_host set to the bad compute node, which is used to start the instance.

I have verified that all my filters on the controller are turned on, and I can see that it is able to get the host details like available RAM, disk etc, but the compute is not running on the host.

From my client script, I have no way of figuring out if the host is bad or not. So what is the correct behavior here. The current behavior is definitely not correct, as there is no indication of error except for going to the controller and going through the logs.

Ideally I would like to see that in the hypervisor list, the bad host is not shown up so I don't pick it. Also is there a way manually to remove a compute from the stack in this case.

regards, Behzad

edit retag flag offensive close merge delete


I can tell you that the scheduler reads metrics from db and chooses the host. If the bad host has ever written something, it could be chosen! The scheduler should check the timestamp of the metrics understanding that something is going bad. Can you explain exactly how nova-compute has crashed?

affo gravatar imageaffo ( 2014-10-27 02:42:03 -0600 )edit

In this case, someone accidentally re-imaged the compute node.

bdastur-s gravatar imagebdastur-s ( 2014-10-27 04:17:02 -0600 )edit


affo gravatar imageaffo ( 2014-11-13 04:25:54 -0600 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2014-12-06 07:36:01 -0600

updated 2014-12-07 08:53:05 -0600

You can disable the nova-node, so that the scheduler wont consider it for VM placement.

If you have any custom script, just check if the nova-node is enabled state.

The command is 'nova-manage service list'.

You may get output like

    Binary           Host                Zone      Status     State  Updated_At
nova-compute     compute-01               zone1     enabled    :-)   2014-12-06 13:04:43
nova-compute     compute-02               zone1     disabled   XXX   2014-12-05 21:10:12
edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-10-26 18:39:12 -0600

Seen: 328 times

Last updated: Dec 07 '14