Ask Your Question

Instances with large images sometimes fail to start/spawn

asked 2016-03-17 09:56:54 -0500

timss gravatar image

OpenStack Kilo 2015.1.1 (Mirantis)
Glance with Swift as backend (running on controllers)

When creating instances with a CentOS 6.6 image sized at 1.1 GB it fails consistently for all compute nodes but one. However when using an Ubuntu 14.04 image sized at ~250 MB it works all the time. This setup used to work, so it may have been triggered by a config change or something else.

Looking at the traceback it looks like Glance is having some issues. Tried restarting both the glance-api and glance-registry service but didn't help. Using Horizon or novaclient it initially reports as 'building', and looking at nova --debug boot [..] looks ok (checked API calls using curl).

Not sure what could be wrong, or where to look next.

Logs filtered for readability.


Error from last host: compute1 (compute1): [...] u'RescheduledException: Build of instance <hash> was rescheduled: HTTPInternalServerError (HTTP 500)\n'
Failed to compute_task_build_instances: No valid host was found. Exceeding max scheduling attempts 3 for instance <hash>. [...]
Setting instance to ERROR state.


INFO nova.compute.manager [...] Starting instance...
WARNING nova.compute.resource_tracker [...] Host field should not be set on the instance until resources have been claimed.
WARNING nova.compute.resource_tracker [...] Host field should not be set on the instance until resources have been claimed.
INFO [...] Attempting claim: memory 4096 MB, disk 10 GB
INFO [...] Total memory: ...
INFO [...] memory limit: ...
INFO [...] Total disk: ...
INFO [...] disk limit: ...
INFO [...] Claim succesful
INFO nova.virt.libvirt.driver [...] Creating image
ERROR nova.compute.manager [...] Instance failed to spawn
TRACE nova.compute.manager [...] Traceback (most recent call):


  File "/usr/lib/python2.7/dist-packages/nova/compute/", line 2471, in _build_resources
    yield resources
  File "/usr/lib/python2.7/dist-packages/nova/compute/", line 2343, in _build_and_run_instance
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 2405, in spawn
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 2803, in _create_image
    instance, size, fallback_from_host)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 5930, in _try_fetch_image_cache
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 231, in cache
    *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 480, in create_image
    prepare_template(target=base, max_size=size, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/", line 445, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 221, in fetch_func_sync
    fetch_func(target=target, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 507, in fetch_image
  File "/usr/lib/python2.7/dist-packages/nova/virt/", line 87, in fetch_to_raw
  File "/usr/lib/python2.7/dist-packages/nova/virt/", line 77, in fetch ...
edit retag flag offensive close merge delete


you should check if the libvirt log file gives you some more information

ichavero gravatar imageichavero ( 2016-03-19 13:20:50 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2016-03-18 11:41:45 -0500

updated 2016-03-18 11:55:26 -0500

First issue is running Kilo on CentOS 6.6
Second issue setting Swift on Controllers instead of setting up at least one Separate Swift Node ( even all in one )
I would suggest set up Swift Node ( all in one ) and :-

$ keystone endpoint-list | grep 8080
$ keystone endpoint-delete <id>
$ openstack endpoint create --publicurl "http://New-IP:8080/v1/AUTH_\$(tenant_id)s"  \
          --adminurl "http://New-IP:8080/v1"  \
          --internalurl "http://New-IP:8080/v1/AUTH_\$(tenant_id)s" --region RegionOne swift 
$ service httpd restart
edit flag offensive delete link more


CentOS 6.6 is one of the images registered in Glance, not the distro that Kilo is running on.

Having a complete and seperate Swift install is definitely a better solution and is in the works, but I'm still curious why this is one is now broken. For now, the 6.6 image is disabled (not public).

timss gravatar imagetimss ( 2016-03-18 12:10:41 -0500 )edit

Thanks for the steps on how to replace the Swift endpoint though!

timss gravatar imagetimss ( 2016-03-18 12:12:22 -0500 )edit

Required commands work for Liberty and Kilo I've placed into answer field.

andrew.shvartz gravatar imageandrew.shvartz ( 2016-03-22 03:59:24 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2016-03-17 09:56:54 -0500

Seen: 621 times

Last updated: Mar 18 '16