Ask Your Question

euca-run-instancess produce vm in indefinite spawning state

asked 2010-07-28 14:24:24 -0500

armando-migliaccio gravatar image

I have checked out revision 182 of nova and I was folliwing instructions available at .

I got everything running on a physical machine, however when I try to launch an instance, the instance is stuck in the spawning state as shown by euca-describe-instances (see below). I have copied the output of each node below, hoping that you could point me in the right direction in troubleshooting this. Many thanks!

armandomi@armandomi-cieg:~/openstack/nova/trunk182$ euca-describe-instances RESERVATION r-wpo4fi31 novaproject default INSTANCE i-ze0bnh1q ami-8jcw97ou spawning (novaproject, armandomi-cieg) 0 m1.small 2010-07-28T16:11:41Z

*********LOGGED INFORMATION ********

************NOVA-API ************ root@armandomi-cieg:~/openstack/nova# nova-api ${NOVA_API_ARGS} --nodaemonize --verbose DEBUG:root:Full set of FLAGS: DEBUG:root:storage_availability_zone : nova DEBUG:root:help : None DEBUG:root:cloudpipe_start_port : 12000 DEBUG:root:compute_topic : compute DEBUG:root:keys_path : /home/armandomi/openstack/nova/trunk182/../data/keys DEBUG:root:rabbit_password : guest DEBUG:root:simple_network_bridge : br0 DEBUG:root:public_interface : vlan1 DEBUG:root:network_size : 256 DEBUG:root:vpn_start_port : 1000 DEBUG:root:simple_network_ips : ['', '', ''] DEBUG:root:helpxml : None DEBUG:root:simple_network : True DEBUG:root:aoe_eth_dev : eth0 DEBUG:root:rabbit_port : 5672 DEBUG:root:logfile : None DEBUG:root:last_shelf_id : 279 DEBUG:root:redis_host : DEBUG:root:? : None DEBUG:root:default_instance_type : m1.small DEBUG:root:volume_topic : volume DEBUG:root:public_range : DEBUG:root:use_vpn : True DEBUG:root:working_directory : ./ DEBUG:root:ec2_url : DEBUG:root:storage_dev : /dev/sdb DEBUG:root:s3_port : 3333 DEBUG:root:fake_libvirt : False DEBUG:root:fake_storage : False DEBUG:root:redis_db : 0 DEBUG:root:credential_key_file : pk.pem DEBUG:root:pidfile : DEBUG:root:rabbit_userid : guest DEBUG:root:simple_network_gateway : DEBUG:root:buckets_path : /home/armandomi/openstack/nova/trunk182/nova/../buckets DEBUG:root:instances_path : /home/armandomi/openstack/nova/trunk182/nova/../instances DEBUG:root:storage_name : armandomi-cieg DEBUG:root:node_name : armandomi-cieg DEBUG:root:networks_path : /home/armandomi/openstack/nova/trunk182/nova/../networks DEBUG:root:daemonize : False DEBUG:root:s3_host : DEBUG:root:aoe_export_dir : /var/lib/vblade-persist/vblades DEBUG:root:images_path : /home/armandomi/openstack/nova/trunk182/nova/../images DEBUG:root:cloud_topic : cloud DEBUG:root:verbose : True DEBUG:root:default_kernel : aki-11111 DEBUG:root:vpn_key_suffix : -key DEBUG:root:libvirt_xml_template : /home/armandomi/openstack/nova/trunk182/nova/compute/libvirt.xml.template DEBUG:root:vpn_client_template : /home/armandomi/openstack/nova/trunk182/nova/cloudpipe/client.ovpn.template DEBUG:root:process_pool_size : 4 DEBUG:root:slots_per_shelf : 16 DEBUG:root:volume_group : nova-volumes DEBUG:root:simple_network_broadcast : DEBUG:root:credential_cert_file : cert.pem DEBUG:root:simple_network_netmask : DEBUG:root:simple_network_dns : DEBUG:root:vpn_end_port : 2000 DEBUG:root:report_interval : 10 DEBUG:root:helpshort : None DEBUG:root:ca_file : cacert.pem DEBUG:root:use_intermediate_ca : False DEBUG:root:vpn_image_id : ami-CLOUDPIPE DEBUG:root:public_vlan : 1 DEBUG:root:use_s3 : True DEBUG:root:private_range : DEBUG:root:credentials_template : /home/armandomi/openstack/nova/trunk182/nova/auth/novarc.template DEBUG:root:vpn_ip : DEBUG ...

edit retag flag offensive close merge delete

6 answers

Sort by ยป oldest newest most voted

answered 2010-08-04 09:43:16 -0500

armando-migliaccio gravatar image

Project name and user are the same. disk-raw is an ASCII file showning this:

<html> <head><title>404 - No Such Resource</title></head> <body>

No Such Resource

File not found.

</body> </html>

Whereas the objectstore's log shows this:

2010-07-30 16:48:52+0100 [HTTPChannel,9,] - - [30/Jul/2010:15:48:51 +0000] "GET /_images/ami-3ig6lc4q/image HTTP/1.1" 404 145 "-" "curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/ libidn/1.15"

I think this might be related to this line 'vi +111 trunk/nova/compute/', which does not raise the exception.

edit flag offensive delete link more

answered 2010-07-31 17:20:12 -0500

vishvananda gravatar image

so your disk-raw is not a valid image. Either the image is not correct before it is copied or it is not being downloaded from objectstore correctly. You may need this patch if your user and project name are different: lp:~vishvananda/nova/fix-curl-project

Nova-volume shouldn't be doing anything at this point, but you might check nova-objectstore for errors. If the above doesn't help, your original disk image may be corrupted. Here is a method for grabbing a working image that we used on our first hack-day:

wget -q -O - | tar xSv euca-bundle-image -i vmlinuz-2.6.32-23-server --kernel true euca-bundle-image -i initrd.img-2.6.32-23-server --ramdisk true euca-bundle-image -i ubuntu-lucid.img euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.32-23-server.manifest.xml euca-upload-bundle -b mybucket -m /tmp/initrd.img-2.6.32-23-server.manifest.xml euca-upload-bundle -b mybucket -m /tmp/ubuntu-lucid.img.manifest.xml

kernel=$(euca-register mybucket/vmlinuz-2.6.32-23-server.manifest.xml | awk '{print $2}') ramdisk=$(euca-register mybucket/initrd.img-2.6.32-23-server.manifest.xml | awk '{print $2}') machine=$(euca-register mybucket/ubuntu-lucid.img.manifest.xml | awk '{print $2}') euca-run-instances $machine --kernel $kernel --ramdisk $ramdisk

wait a while

ssh ubuntu@ # password is ubuntu

edit flag offensive delete link more

answered 2010-08-16 15:19:29 -0500

soren gravatar image

I'm rather sure this problem has been solved in never revision of Nova. Can you recheck please? If it's not, we should turn this into a bug report.

edit flag offensive delete link more

answered 2010-08-18 14:50:59 -0500

armando-migliaccio gravatar image

I was no longer able to replicate this, so I'd imagine that it has been fixed!


edit flag offensive delete link more

answered 2010-07-29 00:48:49 -0500

vishvananda gravatar image

i would first check your instances directory for i-ze0bnh1q dir you should have the following files: libvirt.xml disk disk-raw kernel ramdisk

once the instance actually starts you should see a console.log check the file sizes to see if they are reasonable. If any are missing/zero/very small then compute has messed up the download of the images from objectstore Also check nova-compute.log for exceptions. Sometimes they don't show up in the console output

next, check the /var/log/libvirt/qemu/i-ze0bnh1q.log file to see if it exists and has any useful error message

finally try virsh create libvirt.xml from thie instances/i-ze0bnh1q directory and see if you get an error there.

Hopefully one of these things will lead to some more information and we can see what is going wrong.

edit flag offensive delete link more

answered 2010-07-29 13:58:58 -0500

armando-migliaccio gravatar image

Many thanks for the reply, that did point me in the right direction!! The instance directory shows all files but 'disk'; /var/log/libvirt/qemu/ is empty. When I look at the nova-compute.log there's the following error message:

DEBUG:root:Failed to mount filesystem: mount: you must specify the filesystem type

The nova-volume.log does not show any errors and when I try virsh, I clearly get the following error:

error: Failed to create domain from libvirt.xml error: cannot open file '.../nova/data/instances/<instance-id>/disk': No such file or directory

Any clues? Thanks a lot!

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2010-07-28 14:24:24 -0500

Seen: 339 times

Last updated: Aug 18 '10