Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Launch instance from dashboard throws "Error: No valid host was found."

When trying to launch an instance of configuration 8VCPUs and 32GB RAM I got dashboard error

Error: Failed to launch instance "SMLAB_1": Please try again later [Error: No valid host was found. ].

I have 16 VCPUs and 45 GB RAM available in 1 of the 7 compute nodes with plenty of ephemeral storage as seen from "Hypervisor summary" on dashboard. However, all other nodes are almost full and can not accommodate the instance.

I checked controller log file var/log/nova/scheduler.log which says

2015-12-01 19:23:15.711 7277 ERROR nova.scheduler.filter_scheduler [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Error from last host: node-80.domain.tld (node node-80.domain.tld): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1333, in _build_instance\n
set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 401, in decorated_function\n
return function(self, context, args, kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1751, in _spawn\n
LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n
six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1748, in _spawn\n
block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2296, in spawn\n
block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3712, in _create_domain_and_network\n power_on=power_on)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3613, in _create_domain\n
domain.XMLDesc(0))\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n
six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3608, in _create_domain\n
domain.createWithFlags(launch_flags)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit\n result = proxy_call(self._autowrap, f, *args, *
kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in proxy_call\n rv = execute(f,args,kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker\n rv = meth(args,**kwargs)\n', u' File "/usr/lib64/python2.6/site-packages/libvirt.py", line 947, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', u"libvirtError: internal error: early end of file from monitor: possible problem:\nCannot set up guest memory 'pc.ram': Cannot allocate memory\n\n"] 2015-12-01 19:23:15.769 7277 INFO nova.filters [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] Filter RamFilter returned 0 hosts 2015-12-01 19:23:15.770 7277 WARNING nova.scheduler.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Setting instance to ERROR state.

This is the log of compute node where instance creation was attempted: /var/log/nova/comute.log

[root@node-80 nova]# tail -400 compute.log

2015-12-01 19:17:17.240 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:17:40.850 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:17:40.850 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:17:40.850 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:17:40.886 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:18:06.699 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:18:32.059 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:18:32.060 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:18:32.060 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:18:32.101 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:19:14.025 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:19:38.674 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:19:38.674 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:19:38.675 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:19:38.710 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:20:40.065 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:21:04.330 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:21:04.368 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:21:57.333 3895 AUDIT nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Starting instance... 2015-12-01 19:21:57.519 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Attempting claim: memory 32768 MB, disk 60 GB, VCPUs 8 2015-12-01 19:21:57.520 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total memory: 129013 MB, used: 82432.00 MB 2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] memory limit: 129013.00 MB, free: 46581.00 MB 2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total disk: 1062 GB, used: 390.00 GB 2015-12-01 19:21:57.522 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] disk limit: 1062.00 GB, free: 672.00 GB 2015-12-01 19:21:57.523 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total CPUs: 48 VCPUs, used: 32.00 VCPUs 2015-12-01 19:21:57.524 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] CPUs limit: 96.00 VCPUs, free: 64.00 VCPUs 2015-12-01 19:21:57.525 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Claim successful 2015-12-01 19:21:58.577 3895 INFO nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Creating image 2015-12-01 19:22:06.155 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:22:25.720 3895 WARNING nova.virt.disk.vfs.guestfs [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 13813 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 612 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 8 2015-12-01 19:22:35.090 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:22:48.120 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:23:01.553 3895 ERROR nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] An error occurred while trying to launch a defined domain with xml: <domain type="kvm"> <name>instance-00000083</name> <uuid>417c4039-82ab-4c69-bf8a-dd77afea2674</uuid> <memory unit="KiB">33554432</memory> <currentmemory unit="KiB">33554432</currentmemory> <vcpu placement="static">8</vcpu> <sysinfo type="smbios"> <system> <entry name="manufacturer">Red Hat Inc.</entry> <entry name="product">OpenStack Nova</entry> <entry name="version">2014.1.3-fuel5.1.1.mira5</entry> <entry name="serial">043b2539-f1a7-4dbc-ad52-d4ec4e81cc4b</entry> <entry name="uuid">417c4039-82ab-4c69-bf8a-dd77afea2674</entry> </system> </sysinfo> <os> <type arch="x86_64" machine="pc-i440fx-2.0">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> </features> <cpu mode="host-model"> <model fallback="allow"/> </cpu> <clock offset="utc"> <timer name="pit" tickpolicy="delay"/> <timer name="rtc" tickpolicy="catchup"/> <timer name="hpet" present="no"/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none"/> <source file="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/disk"/> <target dev="vda" bus="virtio"/>

</disk> <controller type="usb" index="0">
</controller> <controller type="pci" index="0" model="pci-root"/> <interface type="bridge"> <mac address="fa:16:3e:ca:6e:b9"/> <source bridge="qbr49525fb4-a7"/> <target dev="tap49525fb4-a7"/> <model type="virtio"/>
</interface> <serial type="file"> <source path="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log"/> <target port="0"/> </serial> <serial type="pty"> <target port="1"/> </serial> <console type="file"> <source path="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log"/> <target type="serial" port="0"/> </console> <input type="tablet" bus="usb"/> <input type="mouse" bus="ps2"/> <input type="keyboard" bus="ps2"/> <graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0" keymap="en-us"> <listen type="address" address="0.0.0.0"/> </graphics> <video> <model type="cirrus" vram="9216" heads="1"/>
</video> <memballoon model="virtio">
</memballoon> </devices> </domain>

2015-12-01 19:23:01.554 3895 ERROR nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Instance failed to spawn

I have googled for < Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas>. But seems like it is not the cause though not sure. Any help is appreciated. Thank you.

Launch instance from dashboard throws "Error: No valid host was found."

When trying to launch an instance of configuration 8VCPUs and 32GB RAM I got dashboard error

Error: Failed to launch instance "SMLAB_1": Please try again later [Error: No valid host was found. ].

I have 16 VCPUs and 45 GB RAM available in 1 of the 7 compute nodes with plenty of ephemeral storage as seen from "Hypervisor summary" on dashboard. However, all other nodes are almost full and can not accommodate the instance.

I checked controller log file var/log/nova/scheduler.log which says

2015-12-01 19:23:15.711 7277 ERROR nova.scheduler.filter_scheduler [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Error from last host: node-80.domain.tld (node node-80.domain.tld): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1333, in _build_instance\n
set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 401, in decorated_function\n
return function(self, context, args, kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1751, in _spawn\n
LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n
six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1748, in _spawn\n
block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2296, in spawn\n
block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3712, in _create_domain_and_network\n power_on=power_on)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3613, in _create_domain\n
domain.XMLDesc(0))\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n
six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3608, in _create_domain\n
domain.createWithFlags(launch_flags)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit\n result = proxy_call(self._autowrap, f, *args, *
kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in proxy_call\n rv = execute(f,args,kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker\n rv = meth(args,**kwargs)\n', u' File "/usr/lib64/python2.6/site-packages/libvirt.py", line 947, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', u"libvirtError: internal error: early end of file from monitor: possible problem:\nCannot set up guest memory 'pc.ram': Cannot allocate memory\n\n"] 2015-12-01 19:23:15.769 7277 INFO nova.filters [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] Filter RamFilter returned 0 hosts 2015-12-01 19:23:15.770 7277 WARNING nova.scheduler.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Setting instance to ERROR state.

This is the log of compute node where instance creation was attempted: /var/log/nova/comute.log

[root@node-80 nova]# tail -400 compute.log

2015-12-01 19:17:17.240 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:17:40.850 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:17:40.850 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:17:40.850 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:17:40.886 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:18:06.699 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:18:32.059 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:18:32.060 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:18:32.060 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:18:32.101 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:19:14.025 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:19:38.674 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:19:38.674 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:19:38.675 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:19:38.710 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:20:40.065 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:21:04.330 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:21:04.368 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:21:57.333 3895 AUDIT nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Starting instance... 2015-12-01 19:21:57.519 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Attempting claim: memory 32768 MB, disk 60 GB, VCPUs 8 2015-12-01 19:21:57.520 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total memory: 129013 MB, used: 82432.00 MB 2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] memory limit: 129013.00 MB, free: 46581.00 MB 2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total disk: 1062 GB, used: 390.00 GB 2015-12-01 19:21:57.522 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] disk limit: 1062.00 GB, free: 672.00 GB 2015-12-01 19:21:57.523 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total CPUs: 48 VCPUs, used: 32.00 VCPUs 2015-12-01 19:21:57.524 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] CPUs limit: 96.00 VCPUs, free: 64.00 VCPUs 2015-12-01 19:21:57.525 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Claim successful 2015-12-01 19:21:58.577 3895 INFO nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Creating image 2015-12-01 19:22:06.155 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:22:25.720 3895 WARNING nova.virt.disk.vfs.guestfs [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 13813 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 612 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 8 2015-12-01 19:22:35.090 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:22:48.120 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:23:01.553 3895 ERROR nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] An error occurred while trying to launch a defined domain with xml: <domain type="kvm"> <name>instance-00000083</name> <uuid>417c4039-82ab-4c69-bf8a-dd77afea2674</uuid> <memory unit="KiB">33554432</memory> <currentmemory unit="KiB">33554432</currentmemory> <vcpu placement="static">8</vcpu> <sysinfo type="smbios"> <system> <entry name="manufacturer">Red Hat Inc.</entry> <entry name="product">OpenStack Nova</entry> <entry name="version">2014.1.3-fuel5.1.1.mira5</entry> <entry name="serial">043b2539-f1a7-4dbc-ad52-d4ec4e81cc4b</entry> <entry name="uuid">417c4039-82ab-4c69-bf8a-dd77afea2674</entry> </system> </sysinfo> <os> <type arch="x86_64" machine="pc-i440fx-2.0">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> </features> <cpu mode="host-model"> <model fallback="allow"/> </cpu> <clock offset="utc"> <timer name="pit" tickpolicy="delay"/> <timer name="rtc" tickpolicy="catchup"/> <timer name="hpet" present="no"/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none"/> <source file="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/disk"/> <target dev="vda" bus="virtio"/>

</disk> <controller type="usb" index="0">
</controller> <controller type="pci" index="0" model="pci-root"/> <interface type="bridge"> <mac address="fa:16:3e:ca:6e:b9"/> <source bridge="qbr49525fb4-a7"/> <target dev="tap49525fb4-a7"/> <model type="virtio"/>
</interface> <serial type="file"> <source path="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log"/> <target port="0"/> </serial> <serial type="pty"> <target port="1"/> </serial> <console type="file"> <source path="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log"/> <target type="serial" port="0"/> </console> <input type="tablet" bus="usb"/> <input type="mouse" bus="ps2"/> <input type="keyboard" bus="ps2"/> <graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0" keymap="en-us"> <listen type="address" address="0.0.0.0"/> </graphics> <video> <model type="cirrus" vram="9216" heads="1"/>
</video> <memballoon model="virtio">
</memballoon> </devices> </domain>

2015-12-01 19:23:01.554 3895 ERROR nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Instance failed to spawn

I have googled for < Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas>. But seems like it is not the cause though not sure. Any help is appreciated. Thank you.

Launch instance from dashboard throws "Error: No valid host was found."

When trying to launch an instance of configuration 8VCPUs and 32GB RAM I got dashboard error

> Error: Failed to launch instance
 > "SMLAB_1": Please try again later
 > [Error: No valid host was found. ].

].

I have 16 VCPUs and 45 GB RAM available in 1 of the 7 compute nodes with plenty of ephemeral storage as seen from "Hypervisor summary" on dashboard. However, all other nodes are almost full and can not accommodate the instance.

I checked controller log file var/log/nova/scheduler.log which says

> 2015-12-01 19:23:15.711 7277 ERROR
 > nova.scheduler.filter_scheduler
 > [req-4985b273-48bf-48f8-9fbe-9a1c62523862
 > None] [instance:
 > 417c4039-82ab-4c69-bf8a-dd77afea2674]
 > Error from last host:
 > node-80.domain.tld (node
 > node-80.domain.tld): [u'Traceback
 > (most recent call last):\n', u'  File
 > "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
 > line 1333, in _build_instance\n 
> set_access_ip=set_access_ip)\n', u' > File > "/usr/lib/python2.6/site-packages/nova/compute/manager.py", > line 401, in decorated_function\n
> return function(self, context, args, kwargs)\n', *args, > **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", > line 1751, in _spawn\n
> LOG.exception(_(\'Instance failed to > spawn\'), instance=instance)\n', u' > File > "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", > line 68, in __exit__\n
> six.reraise(self.type_, self.value, > self.tb)\n', u' File > "/usr/lib/python2.6/site-packages/nova/compute/manager.py", > line 1748, in _spawn\n
> block_device_info)\n', u' File > "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", > line 2296, in spawn\n
> block_device_info)\n', u' File > "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", > line 3712, in > _create_domain_and_network\n power_on=power_on)\n', u' File > "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", > line 3613, in _create_domain\n
> domain.XMLDesc(0))\n', u' File > "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", > line 68, in __exit__\n
> six.reraise(self.type_, self.value, > self.tb)\n', u' File > "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", > line 3608, in _create_domain\n
> domain.createWithFlags(launch_flags)\n', > u' File > "/usr/lib/python2.6/site-packages/eventlet/tpool.py", > line 179, in doit\n result = > proxy_call(self._autowrap, f, *args, *
kwargs)\n', > **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", > line 139, in proxy_call\n rv = execute(f,args,kwargs)\n', > execute(f,*args,**kwargs)\n', u' File > "/usr/lib/python2.6/site-packages/eventlet/tpool.py", > line 77, in tworker\n rv = meth(args,**kwargs)\n', > meth(*args,**kwargs)\n', u' File > "/usr/lib64/python2.6/site-packages/libvirt.py", > line 947, in createWithFlags\n if > ret == -1: raise libvirtError > (\'virDomainCreateWithFlags() > failed\', dom=self)\n', > u"libvirtError: internal error: early > end of file from monitor: possible > problem:\nCannot set up guest memory > 'pc.ram': Cannot allocate memory\n\n"] > 2015-12-01 19:23:15.769 7277 INFO > nova.filters > [req-4985b273-48bf-48f8-9fbe-9a1c62523862 > None] Filter RamFilter returned 0 > hosts 2015-12-01 19:23:15.770 7277 > WARNING nova.scheduler.driver > [req-4985b273-48bf-48f8-9fbe-9a1c62523862 > None] [instance: > 417c4039-82ab-4c69-bf8a-dd77afea2674] > Setting instance to ERROR state.

state.

This is the log of compute node where instance creation was attempted: /var/log/nova/comute.log

[root@node-80 nova]# tail -400 compute.log

2015-12-01 19:17:17.240 compute.log 2015-12-01 19:20:40.065 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:17:40.850 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:17:40.850 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:17:40.850 19:21:04.330 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:17:40.886 19:21:04.368 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:18:06.699 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:18:32.059 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:18:32.060 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:18:32.060 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:18:32.101 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:19:14.025 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:19:38.674 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:19:38.674 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:19:38.675 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:19:38.710 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:20:40.065 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581 2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672 2015-12-01 19:21:04.330 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16 2015-12-01 19:21:04.368 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:21:57.333 3895 AUDIT nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Starting instance... **Starting instance...** 2015-12-01 19:21:57.519 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Attempting claim: memory 32768 MB, disk 60 GB, VCPUs 8 2015-12-01 19:21:57.520 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total memory: 129013 MB, used: 82432.00 MB 2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] memory limit: 129013.00 MB, free: 46581.00 MB 2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total disk: 1062 GB, used: 390.00 GB 2015-12-01 19:21:57.522 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] disk limit: 1062.00 GB, free: 672.00 GB 2015-12-01 19:21:57.523 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total CPUs: 48 VCPUs, used: 32.00 VCPUs 2015-12-01 19:21:57.524 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] CPUs limit: 96.00 VCPUs, free: 64.00 VCPUs 2015-12-01 19:21:57.525 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Claim successful **Claim successful** 2015-12-01 19:21:58.577 3895 INFO nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Creating image 2015-12-01 19:22:06.155 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:22:25.720 3895 WARNING **WARNING** nova.virt.disk.vfs.guestfs [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 13813 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 612 2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 8 2015-12-01 19:22:35.090 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld 2015-12-01 19:22:48.120 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-12-01 19:23:01.553 3895 ERROR **ERROR** nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] An error occurred while trying to launch a defined domain with xml: <domain type="kvm"> type='kvm'> <name>instance-00000083</name> <uuid>417c4039-82ab-4c69-bf8a-dd77afea2674</uuid> <memory unit="KiB">33554432</memory> <currentmemory unit="KiB">33554432</currentmemory> unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <vcpu placement="static">8</vcpu> placement='static'>8</vcpu> <sysinfo type="smbios"> type='smbios'> <system> <entry name="manufacturer">Red name='manufacturer'>Red Hat Inc.</entry> <entry name="product">OpenStack name='product'>OpenStack Nova</entry> <entry name="version">2014.1.3-fuel5.1.1.mira5</entry> name='version'>2014.1.3-fuel5.1.1.mira5</entry> <entry name="serial">043b2539-f1a7-4dbc-ad52-d4ec4e81cc4b</entry> name='serial'>043b2539-f1a7-4dbc-ad52-d4ec4e81cc4b</entry> <entry name="uuid">417c4039-82ab-4c69-bf8a-dd77afea2674</entry> name='uuid'>417c4039-82ab-4c69-bf8a-dd77afea2674</entry> </system> </sysinfo> <os> <type arch="x86_64" machine="pc-i440fx-2.0">hvm</type> arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <boot dev="hd"/> dev='hd'/> <smbios mode="sysinfo"/> mode='sysinfo'/> </os> <features> <acpi/> <apic/> </features> <cpu mode="host-model"> mode='host-model'> <model fallback="allow"/> fallback='allow'/> </cpu> <clock offset="utc"> offset='utc'> <timer name="pit" tickpolicy="delay"/> name='pit' tickpolicy='delay'/> <timer name="rtc" tickpolicy="catchup"/> name='rtc' tickpolicy='catchup'/> <timer name="hpet" present="no"/> name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type="file" device="disk"> type='file' device='disk'> <driver name="qemu" type="qcow2" cache="none"/> name='qemu' type='qcow2' cache='none'/> <source file="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/disk"/> file='/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/disk'/> <target dev="vda" bus="virtio"/>

dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <controller type="usb" index="0">
type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type="pci" index="0" model="pci-root"/> type='pci' index='0' model='pci-root'/> <interface type="bridge"> type='bridge'> <mac address="fa:16:3e:ca:6e:b9"/> address='fa:16:3e:ca:6e:b9'/> <source bridge="qbr49525fb4-a7"/> bridge='qbr49525fb4-a7'/> <target dev="tap49525fb4-a7"/> dev='tap49525fb4-a7'/> <model type="virtio"/>
type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type="file"> type='file'> <source path="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log"/> path='/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log'/> <target port="0"/> port='0'/> </serial> <serial type="pty"> type='pty'> <target port="1"/> port='1'/> </serial> <console type="file"> type='file'> <source path="/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log"/> path='/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log'/> <target type="serial" port="0"/> type='serial' port='0'/> </console> <input type="tablet" bus="usb"/> type='tablet' bus='usb'/> <input type="mouse" bus="ps2"/> type='mouse' bus='ps2'/> <input type="keyboard" bus="ps2"/> type='keyboard' bus='ps2'/> <graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0" keymap="en-us"> type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'> <listen type="address" address="0.0.0.0"/> type='address' address='0.0.0.0'/> </graphics> <video> <model type="cirrus" vram="9216" heads="1"/>
type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model="virtio">
model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> </domain>

</domain> 2015-12-01 19:23:01.554 3895 ERROR nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Instance failed to spawn

spawn

I have googled for < Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas>. But seems like it is not the cause though not sure. Any help is appreciated. Thank you.

Launch instance from dashboard throws "Error: No valid host was found."

When trying to launch an instance of configuration 8VCPUs and 32GB RAM I got dashboard error

> Error: Failed to launch instance
> "SMLAB_1": Please try again later
> [Error: No valid host was found. ].

I have 16 VCPUs and 45 GB RAM available in 1 of the 7 compute nodes with plenty of ephemeral storage as seen from "Hypervisor summary" on dashboard. However, all other nodes are almost full and can not accommodate the instance.

I checked controller log file var/log/nova/scheduler.log which says

> 2015-12-01 19:23:15.711 7277 ERROR
> nova.scheduler.filter_scheduler
> [req-4985b273-48bf-48f8-9fbe-9a1c62523862
> None] [instance:
> 417c4039-82ab-4c69-bf8a-dd77afea2674]
> Error from last host:
> node-80.domain.tld (node
> node-80.domain.tld): [u'Traceback
> (most recent call last):\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 1333, in _build_instance\n   
> set_access_ip=set_access_ip)\n', u' 
> File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 401, in decorated_function\n   
> return function(self, context, *args,
> **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 1751, in _spawn\n   
> LOG.exception(_(\'Instance failed to
> spawn\'), instance=instance)\n', u' 
> File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n   
> six.reraise(self.type_, self.value,
> self.tb)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py",
> line 1748, in _spawn\n   
> block_device_info)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 2296, in spawn\n   
> block_device_info)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 3712, in
> _create_domain_and_network\n    power_on=power_on)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 3613, in _create_domain\n   
> domain.XMLDesc(0))\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n   
> six.reraise(self.type_, self.value,
> self.tb)\n', u'  File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py",
> line 3608, in _create_domain\n   
> domain.createWithFlags(launch_flags)\n',
> u'  File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py",
> line 179, in doit\n    result =
> proxy_call(self._autowrap, f, *args,
> **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",
> line 139, in proxy_call\n    rv =
> execute(f,*args,**kwargs)\n', u'  File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py",
> line 77, in tworker\n    rv =
> meth(*args,**kwargs)\n', u'  File
> "/usr/lib64/python2.6/site-packages/libvirt.py",
> line 947, in createWithFlags\n    if
> ret == -1: raise libvirtError
> (\'virDomainCreateWithFlags()
> failed\', dom=self)\n',
> u"libvirtError: internal error: early
> end of file from monitor: possible
> problem:\nCannot set up guest memory
> 'pc.ram': Cannot allocate memory\n\n"]
> 2015-12-01 19:23:15.769 7277 INFO
> nova.filters
> [req-4985b273-48bf-48f8-9fbe-9a1c62523862
> None] Filter RamFilter returned 0
> hosts 2015-12-01 19:23:15.770 7277
> WARNING nova.scheduler.driver
> [req-4985b273-48bf-48f8-9fbe-9a1c62523862
> None] [instance:
> 417c4039-82ab-4c69-bf8a-dd77afea2674]
> Setting instance to ERROR state.

This is the log of compute node where instance creation was attempted: /var/log/nova/comute.log

[root@node-80 nova]# tail -400 compute.log

2015-12-01 19:20:40.065 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 46581
2015-12-01 19:21:04.329 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 672
2015-12-01 19:21:04.330 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 16
2015-12-01 19:21:04.368 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld
2015-12-01 19:21:57.333 3895 AUDIT nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] **Starting instance...**
2015-12-01 19:21:57.519 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Attempting claim: memory 32768 MB, disk 60 GB, VCPUs 8
2015-12-01 19:21:57.520 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total memory: 129013 MB, used: 82432.00 MB
2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] memory limit: 129013.00 MB, free: 46581.00 MB
2015-12-01 19:21:57.521 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total disk: 1062 GB, used: 390.00 GB
2015-12-01 19:21:57.522 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] disk limit: 1062.00 GB, free: 672.00 GB
2015-12-01 19:21:57.523 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Total CPUs: 48 VCPUs, used: 32.00 VCPUs
2015-12-01 19:21:57.524 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] CPUs limit: 96.00 VCPUs, free: 64.00 VCPUs
2015-12-01 19:21:57.525 3895 AUDIT nova.compute.claims [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] **Claim successful**
2015-12-01 19:21:58.577 3895 INFO nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Creating image
2015-12-01 19:22:06.155 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-12-01 19:22:25.720 3895 **WARNING** nova.virt.disk.vfs.guestfs [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas
2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 13813
2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 612
2015-12-01 19:22:35.051 3895 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 8
2015-12-01 19:22:35.090 3895 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-80.domain.tld:node-80.domain.tld
2015-12-01 19:22:48.120 3895 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-12-01 19:23:01.553 3895 **ERROR** nova.virt.libvirt.driver [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] An error occurred while trying to launch a defined domain with xml: <domain type='kvm'>
  <name>instance-00000083</name>
  <uuid>417c4039-82ab-4c69-bf8a-dd77afea2674</uuid>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <vcpu placement='static'>8</vcpu>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>Red Hat Inc.</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>2014.1.3-fuel5.1.1.mira5</entry>
        <entry name='serial'>043b2539-f1a7-4dbc-ad52-d4ec4e81cc4b</entry>
        <entry name='uuid'>417c4039-82ab-4c69-bf8a-dd77afea2674</entry>
      </system>
    </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='fa:16:3e:ca:6e:b9'/>
      <source bridge='qbr49525fb4-a7'/>
      <target dev='tap49525fb4-a7'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/417c4039-82ab-4c69-bf8a-dd77afea2674/console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

2015-12-01 19:23:01.554 3895 ERROR nova.compute.manager [req-4985b273-48bf-48f8-9fbe-9a1c62523862 None] [instance: 417c4039-82ab-4c69-bf8a-dd77afea2674] Instance failed to spawn

I have googled for < Failed to close augeas aug_close: do_aug_close: you must call 'aug-init' first to initialize Augeas>. But seems like it is not the cause though not sure. Any help is appreciated. Thank you.

Attaching libvirt logs. The last attempt with 16GB RAM is successful.

 2015-12-01 18:10:42.248+0000: 3733: warning : qemuOpenVhostNet:508 : Unable to open vhost-net. Opened so far 0, requested 1
2015-12-01 18:10:42.248+0000: 3733: warning : qemuDomainObjTaint:1658 : Domain id=18 name='instance-00000082' uuid=cb46dd95-5f19-4e21-9258-6523c29c4bc5 is tainted: high-privileges
2015-12-01 18:10:42.619+0000: 3731: error : qemuMonitorIORead:557 : Unable to read from monitor: Connection reset by peer
2015-12-01 18:10:42.619+0000: 3731: error : qemuMonitorIO:694 : internal error: early end of file from monitor: possible problem:
Cannot set up guest memory 'pc.ram': Cannot allocate memory

2015-12-01 19:22:58.647+0000: 3735: warning : qemuOpenVhostNet:508 : Unable to open vhost-net. Opened so far 0, requested 1
2015-12-01 19:22:58.647+0000: 3735: warning : qemuDomainObjTaint:1658 : Domain id=19 name='instance-00000083' uuid=417c4039-82ab-4c69-bf8a-dd77afea2674 is tainted: high-privileges
2015-12-01 19:23:00.631+0000: 3731: error : qemuMonitorIORead:557 : Unable to read from monitor: Connection reset by peer
2015-12-01 19:23:00.631+0000: 3731: error : qemuMonitorIO:694 : internal error: early end of file from monitor: possible problem:
Cannot set up guest memory 'pc.ram': Cannot allocate memory

This is successful with 16GB RAM.

 2015-12-02 18:56:30.327+0000: 3736: warning : qemuOpenVhostNet:508 : Unable to open vhost-net. Opened so far 0, requested 1
    2015-12-02 18:56:30.327+0000: 3736: warning : qemuDomainObjTaint:1658 : Domain id=20 name='instance-00000084' uuid=5d74d4d4-71c7-48e6-bb42-45f848a87b49 is tainted: high-privileges
[root@node-80 libvirt]#