Revision history [back]

click to hide/show revision 1
initial version

Openstack pike: No valid host was found. There are not enough hosts available. - code 500

I have setup a multi nodes Openstack env using tripleO for a PoC. The environment consists of (Running on baremetal):

1- Undercloud
2- Overcloud:
        -Controller
        -Compute-0
        -Compute-1

I have created a few instances and everything is fine but now I'm not able to create any new instance. I have checked the compute resources and they are plenty. I'm using cinder for both storing the instances and images.

 (overcloud) [heat-admin@overcloud-controller-0 ~]$ openstack compute service list
+----+------------------+---------------------------------+-----------+---------+-------+----------------------------+
| ID | Binary           | Host                            | Zone      | Status  | State | Updated At                 |
+----+------------------+---------------------------------+-----------+---------+-------+----------------------------+
|  4 | nova-conductor   | overcloud-controller-0.lab.poc  | internal  | enabled | up    | 2018-03-13T08:50:50.000000 |
| 18 | nova-compute     | overcloud-novacompute-1.lab.poc | compute-1 | enabled | up    | 2018-03-13T08:50:49.000000 |
| 19 | nova-consoleauth | overcloud-controller-0.lab.poc  | internal  | enabled | up    | 2018-03-13T08:50:47.000000 |
| 20 | nova-scheduler   | overcloud-controller-0.lab.poc  | internal  | enabled | up    | 2018-03-13T08:50:51.000000 |
| 21 | nova-compute     | overcloud-novacompute-0.lab.poc | compute-0 | enabled | up    | 2018-03-13T08:50:50.000000 |
+----+------------------+---------------------------------+-----------+---------+-------+----------------------------+
nova-conductor.log (debug enabled):
 2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager [req-c982618d-c1c0-46f1-9182-3e542f5af8af 5c1e3b2d8bc44ca08d3ccfe9c2a6ff17 7e66bf9554f748018600d046f22acc16 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found. There are not enough hosts available. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 149, in select_destinations
    alloc_reqs_by_rp_uuid, provider_summaries)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 110, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager Traceback (most recent call last):
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1036, in schedule_and_build_instances
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     instance_uuids)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 626, in _schedule_instances
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     request_spec, instance_uuids)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 586, in wrapped
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     return func(*args, **kwargs)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 52, in select_destinations
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     instance_uuids)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 33, in select_destinations
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     instance_uuids)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 137, in select_destinations
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     retry=self.retry)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     timeout=timeout, retry=retry)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     retry=retry)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     raise result
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager Traceback (most recent call last):
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     return func(*args, **kwargs)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 149, in select_destinations
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     alloc_reqs_by_rp_uuid, provider_summaries)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 110, in select_destinations
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager     raise exception.NoValidHost(reason=reason)
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager
2018-03-13 07:13:52.676 3840 ERROR nova.conductor.manager
2018-03-13 07:13:53.145 3840 WARNING nova.scheduler.utils [req-c982618d-c1c0-46f1-9182-3e542f5af8af 5c1e3b2d8bc44ca08d3ccfe9c2a6ff17 7e66bf9554f748018600d046f22acc16 - default default] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 149, in select_destinations
    alloc_reqs_by_rp_uuid, provider_summaries)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 110, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 149, in select_destinations
    alloc_reqs_by_rp_uuid, provider_summaries)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 110, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
2018-03-13 07:13:53.146 3840 WARNING nova.scheduler.utils [req-c982618d-c1c0-46f1-9182-3e542f5af8af 5c1e3b2d8bc44ca08d3ccfe9c2a6ff17 7e66bf9554f748018600d046f22acc16 - default default] [instance: 8e236166-d42e-4255-bda1-a3e7fcbae75c] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 149, in select_destinations
    alloc_reqs_by_rp_uuid, provider_summaries)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 110, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available. 

nova-scheduler.log:

 2018-03-13 07:13:52.673 2088 INFO nova.filters [req-c982618d-c1c0-46f1-9182-3e542f5af8af 5c1e3b2d8bc44ca08d3ccfe9c2a6ff17 7e66bf9554f748018600d046f22acc16 - default default] Filter RetryFilter returned 0 hosts
2018-03-13 07:13:52.673 2088 INFO nova.filters [req-c982618d-c1c0-46f1-9182-3e542f5af8af 5c1e3b2d8bc44ca08d3ccfe9c2a6ff17 7e66bf9554f748018600d046f22acc16 - default default] Filtering removed all hosts for the request with instance ID '8e236166-d42e-4255-bda1-a3e7fcbae75c'. Filter results: ['RetryFilter: (start: 0, end: 0)']

 (openstack) hypervisor stats show
+----------------------+--------+
| Field                | Value  |
+----------------------+--------+
| count                | 3      |
| current_workload     | 0      |
| disk_available_least | 799    |
| free_disk_gb         | 404    |
| free_ram_mb          | 314692 |
| local_gb             | 834    |
| local_gb_used        | 430    |
| memory_mb            | 393028 |
| memory_mb_used       | 78336  |
| running_vms          | 6      |
| vcpus                | 100    |
| vcpus_used           | 21     |
+----------------------+--------+

I'm at my wit's end. I have been trying to find a fix for this and it has been a few days now and no luck. Please help. Thank you.