Using NFS but overview shows high "local storage" use and get no valid host launching instance.

asked 2018-03-20 09:41:55 -0500

MattF gravatar image

updated 2018-03-20 09:51:38 -0500

I'm having an issue where I tried to launch 10 instances that have 160GB disks.

I have 6 compute instances, and very little running on them apart from about 4/5 test machines. I have 10TB assigned via netapp NFS server and it seems to be working ok.

The problem came today , when I tried to launch 10 instances. It launched 4 of them and stopped with the "no valid host" message. The logs are really unhelpful and this is all there is ...

nova-scheduler.log:2018-03-20 14:20:08.556 2299 INFO nova.filters [req-4654ed16-cb96-4878-a6d2-cfff7ea8703d 8dca0f7fabf94914b4d02f040792e18c 71792c2c288f441c91323049918e327c - default default] Filtering removed all hosts for the request with instance ID 'a9e92da4-642a-499e-9ac5-472f6f5a0d57'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'RamFilter: (start: 1, end: 1)', 'DiskFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 0)']

There's nothing else to go on in the logs except that.

So anyway, I decided to delete all my test VMs and now I am able to launch more instances. Yet I have over 9TB of free space on the NFS server. But what bothers me too is that if I go into Horizon, and I go to Admin > Compute > Hypervisors it was showing very high "local storage" use. Like 1.6TB out of 2TB.

Here is what it says now:

ah I can't post a link (not enough karma) anyway I was trying to post an image of it saying 1.1TB of 2TB local storage.

Where is it getting this 2TB limit from ??! And why ? I'm using a 10TB NFS. The local storage isn't decreasing

But why is this ? I'm not using local storage. In fact if I look at all of my compute nodes disk space I see :

/dev/mapper/centos-root  441G  4.1G  437G   1% /
devtmpfs                 252G     0  252G   0% /dev
tmpfs                    252G     0  252G   0% /dev/shm
tmpfs                    252G   18M  252G   1% /run
tmpfs                    252G     0  252G   0% /sys/fs/cgroup
/dev/sda1               1014M  191M  824M  19% /boot   9.5T  310G  9.2T   4% /var/lib/nova/mnt/8ee65072f6783845e3a3387891518c82
tmpfs                     51G     0   51G   0% /run/user/0

So the compute nodes have over 437GB of local space... and as you can see the NFS has over 9TB free. So why is openstack saying I have 2TB of local disk space and it's nearly all used?

How can I correct this so that it does not limit me because of "local storage"

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2018-03-21 06:13:47 -0500

Deepa gravatar image

I guess the 2TB should be of the total internal disk size(local storage) of your 6 compute node and when you tried to launch 10 instance with 160 GB each (160*10 =1600Gb ~ 1.6 TB) which was not successful yet updated the database . Make sure to delete all the volumes and as a last resort update DB

edit flag offensive delete link more


Thanks for your answer. But why would it try to use space from the local storage when I'm using netapp remote NFS server? The amount of free local disk space is not decreasing. The NFS used space is increasing. So why would it not be successful when I don't use local storage for my instances Thanks

MattF gravatar imageMattF ( 2018-03-21 08:20:09 -0500 )edit

Ran out of comment space. But I think you are right about where the 2TB comes from. It does match the amount of disk space available on all the compute nodes combined. I just can't get why it think im using it when I'm using NFS.

MattF gravatar imageMattF ( 2018-03-21 08:22:36 -0500 )edit

@MattF ,In overview of Horizon NFS or any external storage space wont get reflected,even if we launch instance from it .Local storage doesn't represent the used volume size of external disk.And as your Database entry is wrong ,you may need to manually update volume under Project in DB .

Deepa gravatar imageDeepa ( 2018-03-21 23:57:11 -0500 )edit

If you need help with commands on updating DB let me know Good luck !

Deepa gravatar imageDeepa ( 2018-03-21 23:58:27 -0500 )edit

Hi Deepak. Sorry for my late reply... Yes can you provide the commands you think I need to run? I'm lost at this point Thanks very much.

MattF gravatar imageMattF ( 2018-03-26 10:07:31 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2018-03-20 09:41:55 -0500

Seen: 157 times

Last updated: Mar 21 '18