Launch instance from dashboard throws "Error: No valid host was found."

asked 2015-12-01 09:12:26 -0600

Rajesh@SIT gravatar image

updated 2015-12-02 07:51:36 -0600

When trying to launch an instance of configuration 8VCPUs and 32GB RAM I got dashboard error

> Error: Failed to launch instance
> "SMLAB_1": Please try again later
> [Error: No valid host was found. ].

I have 16 VCPUs and 45 GB RAM available in 1 of the 7 compute nodes with plenty of ephemeral storage as seen from "Hypervisor summary" on dashboard. However, all other nodes are almost full and can not accommodate the instance.

I checked controller log file var/log/nova/scheduler.log which says

> 2015-12-01 19:23:15.711 7277 ERROR
> nova.scheduler.filter_scheduler
> [req-4985b273-48bf-48f8-9fbe-9a1c62523862
> None] [instance:
> 417c4039-82ab-4c69-bf8a-dd77afea2674]
> Error from last host:
> node-80.domain.tld (node
> node-80.domain.tld): [u'Traceback
> (most recent call last):\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 1333, in _build_instance\n   
> set_access_ip=set_access_ip)\n', u' 
> File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 401, in decorated_function\n   
> return function(self, context, *args,
> **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 1751, in _spawn\n   
> LOG.exception(_(\'Instance failed to
> spawn\'), instance=instance)\n', u' 
> File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n   
> six.reraise(self.type_, self.value,
> self.tb)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 1748, in _spawn\n   
> block_device_info)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 2296, in spawn\n   
> block_device_info)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 3712, in
> _create_domain_and_network\n    power_on=power_on)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 3613, in _create_domain\n   
> domain.XMLDesc(0))\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n   
> six.reraise(self.type_, self.value,
> self.tb)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 3608, in _create_domain\n   
> domain.createWithFlags(launch_flags)\n',
> u'  File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py",
> line 179, in doit\n    result =
> proxy_call(self._autowrap, f, *args,
> **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",
> line 139, in proxy_call\n    rv =
> execute(f,*args,**kwargs)\n', u'  File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py",
> line 77, in tworker\n    rv =
> meth(*args,**kwargs)\n', u'  File
> "/usr/lib64/python2.6/site-packages/libvirt.py",
> line 947, in createWithFlags\n    if
> ret == -1: raise libvirtError
> (\'virDomainCreateWithFlags()
> failed\', dom=self)\n',
> u"libvirtError: internal error: early
> end of file from monitor: possible
> problem:\nCannot set up guest memory
> 'pc.ram': Cannot allocate memory\n\n"]
> 2015-12-01 19:23:15.769 7277 INFO
> nova.filters
> [req-4985b273-48bf-48f8-9fbe-9a1c62523862
> None] Filter RamFilter returned 0
> hosts 2015-12-01 19:23:15.770 7277
> WARNING nova.scheduler.driver
> [req-4985b273-48bf-48f8-9fbe-9a1c62523862
> None] [instance:
> 417c4039-82ab-4c69-bf8a-dd77afea2674]
> Setting instance to ERROR state.

This is the log of compute node where instance creation was attempted: /var/log/nova/comute.log ... (more)

edit retag flag offensive close merge delete

Comments

  1. check the status of nova process on compute.
  2. problems seems to be allocation of memory on compute.
  3. share the result of creation of an instance of medium flavor , it will clarify the exact cause.
  4. please share libvirt logs as well.
ritesh.singh.aricent@gmail.com gravatar imageritesh.singh.aricent@gmail.com ( 2015-12-02 02:08:39 -0600 )edit

Hi Ritesh,

  1. Creation of a smaller instance with 16 GB RAM is successful (on the same compute node).
  2. Added libvirt logs above. Now I wonder why my other 6 compute nodes can reach its full capacity (i.e >120GB of 125 available) when this particular node can not. Thank you.
Rajesh@SIT gravatar imageRajesh@SIT ( 2015-12-02 07:46:30 -0600 )edit

Can you check the following - Under Admin tab - check the Hypervisors and see what does it report as RAM (used) and RAM (total) - This should tell us what is the hypervisor status.

Check for this link :- it might be of help http://earlruby.org/2014/05/increase-...

ritesh.singh.aricent@gmail.com gravatar imageritesh.singh.aricent@gmail.com ( 2015-12-02 21:59:54 -0600 )edit

Hi, RAM total is 125GB and used is 80 GB before launching the instance in the said compute node. So, it has 45 GB of physical RAM available. My other nodes reached 124 GB s. Is there any node specific configuration to look that prevents it allocating- e.g in nova.conf? I'm new to OpenStack. Thanks.

Rajesh@SIT gravatar imageRajesh@SIT ( 2015-12-02 23:29:46 -0600 )edit