Ask Your Question

havana - cannot spawn instance

asked 2013-11-04 08:06:18 -0500

senyapsudah gravatar image

Hi Guys,

i'm currently trying to bring up my openstack server. there are two servers. 1 running as a controller( most of the api) and another as compute(nova compute and cinder). i manage to get dashboard and able to create image, bootable volume and others. but the problem is i'm not able to bring up my VM on. always end up with error.

below are the error that i'm able to find in my nova-scheduler.log.

2013-11-04 15:45:51.262 7130 ERROR nova.scheduler.filter_scheduler [req-87fa924d-c634-4a37-acc0-6746d7150010 a789942604184ebf9b6fac85b279b76b 7666f2651b3c440a9b1c3ea0c8cf23d2] [instance: 8c84f0c3-4e70-4862-bee4-b8c2fbaaa017] Error from l
ast host: Stack-Compute (node Stack-Compute): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/", line 1037, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/u
sr/lib/python2.7/dist-packages/nova/compute/", line 1410, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/", line 1407, in _
spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 2063, in spawn\n    admin_pass=admin_password)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", l
ine 2353, in _create_image\n    project_id=instance[\'project_id\'])\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 174, in cache\n    *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packag
es/nova/virt/libvirt/", line 276, in create_image\n    prepare_template(target=base, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/", line 246, in inner\n    return f(*ar
gs, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 162, in call_if_not_exists\n    fetch_func(target=target, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/lib
virt/", line 644, in fetch_image\n    images.fetch_to_raw(context, image_id, target, user_id, project_id)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/", line 195, in fetch_to_raw\n    fetch(context, image_hr
ef, path_tmp, user_id, project_id)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/", line 190, in fetch\n, image_id, dst_path=path)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/
image/", line 360, in download\n    for chunk in image_chunks:\n', u'  File "/usr/lib/python2.7/dist-packages/glanceclient/common/", line 478, in __iter__\n    chunk =\n', u'  File "/usr/lib/python2.7/dist-pac
kages/glanceclient/common/", line 494, in next\n    chunk =\n', u'  File "/usr/lib/python2.7/", line 561, in read\n    s =\n', u'  File "/usr/lib/python2.7/", line 38
0, in read\n    data = self._sock.recv(left)\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/", line 262, in recv\n    timeout_exc=socket ...
edit retag flag offensive close merge delete


Do you see anything else in your compute log (`/var/log/nova/compute.log`) that corresponds to this failure? It looks like the compute service may be having issues communicating with `glance`. From the compute host, are you able to connect to `glance` using curl? If you run `curl http://controller-address:9292` you should get back a JSON reply that begins with `{"versions": ...`.

larsks gravatar imagelarsks ( 2013-11-04 09:30:50 -0500 )edit

Yep, i have no issue to connect to glance from compute note using curl. but after i enable debug in nova, i found out the issue is cause by cannot attached volume using tgt. as inside the logs it stated connection using iscsiadm. after i change to ietadm and stop tgt there no more error on that.

senyapsudah gravatar imagesenyapsudah ( 2013-11-04 23:04:31 -0500 )edit

however, now other issue coming up. seems like i got a problem with libvirt. "'libvirtError: Unable to read from monitor: Connection reset by peer\n'". i have check inside the logs and found a lot of errors. virNetSocketReadWire:1377 : End of file while reading data: Input/output error. idea?

senyapsudah gravatar imagesenyapsudah ( 2013-11-04 23:10:09 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2013-11-05 00:37:02 -0500

senyapsudah gravatar image

After crack all around the place.. finally found the issue.. there is a few things that need to be monitored..

  1. monitor the syslog: here i found out below error

    apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/libvirtd" pid=2217 comm="libvirtd" pid=2217 comm="libvirtd" capability=29 capname="audit_write"

seems like apparmor is blocking libvirt from running script. the solution is to allow libvirt running "audit_write". you can do this by adding into this fil /etc/apparmor.d/usr.sbin.libvirtd. syntax is

capability audit_write,

then found issue on vnc not able to run. need to correct the /etc/nova/nova.conf on compute server. to add below line.


the above line will allow vnc to listen on compute ip address and you can access the VM console from horizon. seems like everything fine now. :)

but i belive there will be a lot more to go as still in the process of getting ceilometer and swift to work.

thanks for all the support guys.

edit flag offensive delete link more


there is also a way which is much more easier, where you can force all apparmor profiles to complain mode. you may do this by installing apparmor-utils and you may use aa-complaint /etc/apparmod.d/* or you may totally disable apparmor from your system. but make sure that your libvirt does not use apparmor as a security module.

senyapsudah gravatar imagesenyapsudah ( 2014-04-27 18:48:12 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2013-11-04 08:06:18 -0500

Seen: 6,537 times

Last updated: Nov 05 '13