Ask Your Question
0

Liberty: NoValidHost: No valid host was found. There are not enough hosts available.

asked 2015-12-10 10:42:27 -0600

Herr-Herner gravatar image

I am able to launch "small" instances that require only a small disk, but spawning instances with larger disk fails. We are using a 1 TB shared storage mounted into all 3 compute nodes via NFS. When "exceeding" the disk limit, we get the following exception:

NoValidHost: No valid host was found. There are not enough hosts available.

It seems that all three compute nodes are filtered out by nova:

Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70
Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
(mos-compute02, mos-compute02.localdomain) ram:29103 disk:13312 io_ops:0 instances:2 does not have 81920 MB usable disk, it only has 13312.0 MB usable disk. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/disk_filter.py:60
(mos-compute01, mos-compute01.localdomain) ram:29611 disk:14336 io_ops:0 instances:1 does not have 81920 MB usable disk, it only has 14336.0 MB usable disk. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/disk_filter.py:60
(mos-compute03, mos-compute03.localdomain) ram:29103 disk:13312 io_ops:0 instances:2 does not have 81920 MB usable disk, it only has 13312.0 MB usable disk. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/disk_filter.py:60
Filter DiskFilter returned 0 hosts
Filtering removed all hosts for the request with reservation ID 'r-wyf01la7' and instance ID '4035e2b9-1ccc-454e-b5a5-e0b785e78c92'. Filter results: [('RetryFilter', [(u'mos-compute02', u'mos-compute02.localdomain'), (u'mos-compute01', u'mos-compute01.localdomain'), (u'mos-compute03', u'mos-compute03.localdomain')]), ('AvailabilityZoneFilter', [(u'mos-compute02', u'mos-compute02.localdomain'), (u'mos-compute01', u'mos-compute01.localdomain'), (u'mos-compute03', u'mos-compute03.localdomain')]), ('RamFilter', [(u'mos-compute02', u'mos-compute02.localdomain'), (u'mos-compute01', u'mos-compute01.localdomain'), (u'mos-compute03', u'mos-compute03.localdomain')]), ('DiskFilter', None)] get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:122
Filtering removed all hosts for the request with reservation ID 'r-wyf01la7' and instance ID '4035e2b9-1ccc-454e-b5a5-e0b785e78c92'. Filter results: ['RetryFilter: (start: 3, end: 3)', 'AvailabilityZoneFilter: (start: 3, end: 3)', 'RamFilter: (start: 3, end: 3)', 'DiskFilter: (start: 3, end: 0)']
There are 0 hosts available but 1 instances requested to build. select_destinations /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:87

nova hypervisor-stats:

+----------------------+-------+
| Property             | Value |
+----------------------+-------+
| count                | 3     |
| current_workload     | 0     |
| disk_available_least | 40    |
| free_disk_gb         | 52    |
| free_ram_mb          | 87817 |
| local_gb             | 114   |
| local_gb_used        | 62    |
| memory_mb            | 96521 |
| memory_mb_used       | 8704  |
| running_vms          | 5     |
| vcpus                | 24    |
| vcpus_used           | 5     |
+----------------------+-------+

The hypervisor tells Nova:

Hypervisor/Node resource view: name=mos-compute01.localdomain free_ram=31216MB free_disk=36GB ...

I think the free_disk is the available physical space on the system disk of the compute node, but how can I tell Nova that the available space for instance disk files is about 1 TB?

edit retag flag offensive close merge delete

Comments

Whats the output of the following ?

grep instances_path /etc/nova/nova.conf  (Compute Node)

Regards

Bipin gravatar imageBipin ( 2015-12-12 22:06:03 -0600 )edit

What I can see is by default the nova instances are made in /var/lib/nova/instances under compute node.... Now does this folder has so much of space ... use df -h ... or best thing is try this under the 'instances path' 'fallocate -l 1024GB myfile' .. This will create a dummy file of 1024GB and test

Prateek K gravatar imagePrateek K ( 2015-12-17 04:33:41 -0600 )edit

grep instances_path /etc/nova/nova.conf (Compute Node) returns instances_path = $state_path/instances

Herr-Herner gravatar imageHerr-Herner ( 2015-12-17 23:31:30 -0600 )edit

Hi. I have configured Openstack using Juno. I am facing the exact same problem. In my nova.conf file there is no variable as instances_path its state_path. So this do not produce any output. While launching Ubuntu instances, its saying /root/dev : no such file or directory

zekken gravatar imagezekken ( 2015-12-21 09:28:56 -0600 )edit

If there is no such variable edit the file and insert this variable in the nova.conf and restart the services and see if it works

Prateek K gravatar imagePrateek K ( 2015-12-21 22:52:41 -0600 )edit

2 answers

Sort by ยป oldest newest most voted
0

answered 2015-12-17 23:49:27 -0600

Herr-Herner gravatar image

updated 2015-12-18 00:16:10 -0600

This is the output of df -h:

Filesystem                     Size  Used Avail Use% Mounted on
udev                            16G  4.0K   16G   1% /dev
tmpfs                          3.2G  772K  3.2G   1% /run
/dev/sde5                       60G  3.2G   54G   6% /
none                           4.0K     0  4.0K   0% /sys/fs/cgroup
none                           5.0M     0  5.0M   0% /run/lock
none                            16G     0   16G   0% /run/shm
none                           100M     0  100M   0% /run/user
/dev/sde1                      184M   40M  136M  23% /boot
mos-storage02:/nova/instances   39G  4.0G   33G  11% /var/lib/nova/instances

That does not seem to be right. It gives me a free disk space of 33G.

Executing df -a -h on mos-storage02gives me:

/dev/mapper/system-root   39G  4.0G   33G  11% /export/nova/instances

Oh, no! I think, I have not mounted the right LVM volume /dev/storage/nfs to /var/lib/nova/instances, but exported the folder on the system disk. I will check this.

edit flag offensive delete link more

Comments

You have your first clue... You can debug it further :)

Prateek K gravatar imagePrateek K ( 2015-12-17 23:56:02 -0600 )edit

/etc/fastab on mos-storage01 was missing a mount entry. Everything is working now! Thank you very much for your help!

Herr-Herner gravatar imageHerr-Herner ( 2015-12-18 00:51:19 -0600 )edit

What exactly have you done? Please tell as I am facing a similar issue

zekken gravatar imagezekken ( 2015-12-22 09:13:36 -0600 )edit

I am using a shared storage mounted via NFS in all 3 compute nodes. My storage node has a partition of 950 GB, but I had forgotten to mount this partition locally and just exported the location on the system partition.

Herr-Herner gravatar imageHerr-Herner ( 2015-12-23 10:37:47 -0600 )edit

Please make an isolated post, describe your environment and tell us what you have done. Post your nova.conf. Do you use a shared storage? What's the output of nova hypervisor-stats? Set debug = true in your nova.conf and restart all nova services. Check the log and try to spawn an instance.

Herr-Herner gravatar imageHerr-Herner ( 2015-12-23 10:40:00 -0600 )edit
0

answered 2015-12-17 02:52:47 -0600

Vinoth gravatar image

Hi,

Check whether NFS is mounted to the correct path.

Above log shows unusable Disk which means Disk is already overcommitted. So increase the overcommit disk ratio.

Thanks,

Vinoth

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-12-10 10:42:27 -0600

Seen: 4,331 times

Last updated: Dec 18 '15