Ask Your Question
0

Getting unexpected NoValidHost: No valid host was found. There are not enough hosts available. error

asked 2017-07-24 18:55:30 -0600

eswenson gravatar image

I'm running openstack ocata. Any attempt to launch any flavor and any instance results in the error: "No valid host was found. There are not enough hosts available." Examining nova-schedule.log on the control shows:

2017-07-24 02:38:56.046 14400 WARNING nova.scheduler.host_manager [req-498bfbb6-8940-4706-8322-9c893ba61763 9879cf1dc8954d5a9606fb555beed1fb f2d743585bd84323959131e1aabd885b - - -] No compute service record found for host ip-10-0-0-\
180
2017-07-24 02:38:56.047 14400 WARNING nova.scheduler.host_manager [req-498bfbb6-8940-4706-8322-9c893ba61763 9879cf1dc8954d5a9606fb555beed1fb f2d743585bd84323959131e1aabd885b - - -] No compute service record found for host ip-10-0-0-\
78
2017-07-24 02:38:56.047 14400 INFO nova.filters [req-498bfbb6-8940-4706-8322-9c893ba61763 9879cf1dc8954d5a9606fb555beed1fb f2d743585bd84323959131e1aabd885b - - -] Filter RetryFilter returned 0 hosts
2017-07-24 02:38:56.047 14400 INFO nova.filters [req-498bfbb6-8940-4706-8322-9c893ba61763 9879cf1dc8954d5a9606fb555beed1fb f2d743585bd84323959131e1aabd885b - - -] Filtering removed all hosts for the request with instance ID '4ed4c17\
9-22ad-4eaf-a417-3deebc6ccd06'. Filter results: ['RetryFilter: (start: 0, end: 0)']

Running "nova hypervisor-list" shows:

+----+---------------------------------------------+-------+---------+
| ID | Hypervisor hostname                         | State | Status  |
+----+---------------------------------------------+-------+---------+
| 3  | ip-10-0-0-180.eu-central-1.compute.internal | up    | enabled |
| 4  | ip-10-0-0-78.eu-central-1.compute.internal  | up    | enabled |
+----+---------------------------------------------+-------+---------+

Running "nova service-list" shows:

+----+------------------+---------------------------------------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host                                        | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+---------------------------------------------+----------+---------+-------+----------------------------+-----------------+
| 9  | nova-cert        | ip-10-0-0-92.eu-central-1.compute.internal  | internal | enabled | up    | 2017-07-24T23:47:31.000000 | -               |
| 10 | nova-consoleauth | ip-10-0-0-92.eu-central-1.compute.internal  | internal | enabled | up    | 2017-07-24T23:47:36.000000 | -               |
| 11 | nova-conductor   | ip-10-0-0-92.eu-central-1.compute.internal  | internal | enabled | up    | 2017-07-24T23:47:40.000000 | -               |
| 12 | nova-scheduler   | ip-10-0-0-92.eu-central-1.compute.internal  | internal | enabled | up    | 2017-07-24T23:47:34.000000 | -               |
| 13 | nova-compute     | ip-10-0-0-180.eu-central-1.compute.internal | nova     | enabled | up    | 2017-07-24T23:47:31.000000 | -               |
| 14 | nova-compute     | ip-10-0-0-78.eu-central-1.compute.internal  | nova     | enabled | up    | 2017-07-24T23:47:36.000000 | -               |
+----+------------------+---------------------------------------------+----------+---------+-------+----------------------------+-----------------+

The two compute instances are up and running and have plenty of memory/disk/etc. I have never been successful in running even the tiniest of instance. The errors are always the same.

In nova-condutor.log, I see this:

2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager [req-498bfbb6-8940-4706-8322-9c893ba61763 9879cf1dc8954d5a9606fb555beed1fb f2d743585bd84323959131e1aabd885b - - -] Failed to schedule instances
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager Traceback (most recent call last):
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 866, in schedule_and_build_instances
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager     request_specs[0].to_legacy_filter_properties_dict())
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 597, in _schedule_instances
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager     hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager     return func(*args, **kwargs)
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager     return self.queryclient.select_destinations(context, spec_obj)
2017-07-24 02:38:56.062 14437 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-07-24 02:38:56.062 14437 ...
(more)
edit retag flag offensive close merge delete

Comments

The compute nodes are unable to update their resource records. Thus the scheduler doesn't know what resources (CPU, RAM, Disk) they have and gives up.

Resource records are in the Placement Service. Run nova-status upgrade check to confirm that something is wrong there.

Bernd Bausch gravatar imageBernd Bausch ( 2017-07-24 20:28:09 -0600 )edit

In the nova-compute.log files, look for messages about not being able to reach the placement service. If so, check network, firewall, configuration etc. to understand what prevents the compute nodes from reaching the placement service.

Bernd Bausch gravatar imageBernd Bausch ( 2017-07-24 20:29:21 -0600 )edit

I had done that and did it again. "nova-status upgrade check" shows that everything is good.

eswenson gravatar imageeswenson ( 2017-07-24 22:01:54 -0600 )edit

even i am facing same issue but i tried with nova-status upgrade check, its working properly. Can you please go through the question/query raised and please help me out

Question raised:- Ocata Ubuntu 16.04 Openstack installation: Getting manager creation issue in neutron while launching an instance

Ruchir Nerurkar gravatar imageRuchir Nerurkar ( 2017-07-26 19:03:54 -0600 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2017-07-25 17:08:02 -0600

eswenson gravatar image

The issue turned out to be this: The compute and controller nodes had FQDNs configured for their hostnames. The contents of /etc/hostname, and the value returned by hostname were FQDNs. This appears to be a Centos/RHEL default, or perhaps the result of bring up a Centos instance on AWS/EC2. In any case, Nova gets confused in this case and when trying to find a host on which to launch an instance, looks for entries in the various Nova database tables with the host name up to and excluding the first "." in the name. We ended up fixing this by correcting the hostnames on all instances, stopping all nova/neutron-related services, dropping all the SQL tables for nova, nova_api, and nova_cell0, re-syncing the api_db and db, and then restarting all services.

edit flag offensive delete link more

Comments

even i am facing same issue but i tried with nova-status upgrade check, its working properly. Can you please go through the question/query raised and please help me out Question raised:- Ocata Ubuntu 16.04 Openstack installation: Getting manager creation issue in neutron while launching an instance

Ruchir Nerurkar gravatar imageRuchir Nerurkar ( 2017-07-26 19:20:14 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2017-07-24 18:55:30 -0600

Seen: 1,138 times

Last updated: Jul 25