Ask Your Question
0

euca-run-instances: failed to spawn

asked 2011-03-17 10:59:45 -0500

cuonghoangce gravatar image

When launching an instance, I get the instances failing to spawn

INSTANCE i-00000013 ami-57e5e221 10.0.0.21 10.0.0.21 failed to spawn openstack (project203, ubuntu2) 0 m1.tiny 2011-03-17T09:42:55Z nova

File nova-compute.log:

2011-03-17 16:43:04,293 INFO nova.virt.libvirt_conn [-] instance instance-00000013: Creating image 2011-03-17 16:43:04,351 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/13f6980d /var/lib/nova/instances/instance-00000013/kernel from (pid=2431) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 16:43:04,375 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/57e5e221_sm /var/lib/nova/instances/instance-00000013/disk from (pid=2431) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 16:43:04,552 INFO nova.virt.libvirt_conn [-] instance instance-00000013: injecting key into image 1474683425 2011-03-17 16:43:04,562 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-00000013/disk from (pid=2431) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 16:43:05,602 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/nbd15 from (pid=2431) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 16:43:05,629 DEBUG nova.utils [-] Result was 1 from (pid=2431) execute /usr/lib/pymodules/python2.6/nova/utils.py:161 2011-03-17 16:43:05,629 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -d /dev/nbd15 from (pid=2431) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 16:43:05,643 WARNING nova.virt.libvirt_conn [-] instance instance-00000013: ignoring error injecting data into image 1474683425 (Unexpected error while running command. Command: sudo tune2fs -c 0 -i 0 /dev/nbd15 Exit code: 1 Stdout: 'tune2fs 1.41.11 (14-Mar-2010)\n' Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\r\nCouldn't find valid filesystem superblock.\n") 2011-03-17 16:43:06,723 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 425, in spawn (nova.exception): TRACE: self._conn.createXML(xml, 0) (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) (nova.exception): TRACE: libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/2 (nova.exception): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000013/kernel': Inappropriate ioctl for device (nova.exception): TRACE: (nova.exception): TRACE: 2011-03-17 16:43:06,735 ERROR nova.compute.manager [H4AWEFSHRNZ4ZSPEGEBN user203 project203] instance 19: Failed to spawn (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules ... (more)

edit retag flag offensive close merge delete

8 answers

Sort by ยป oldest newest most voted
0

answered 2011-03-20 00:46:02 -0500

jo0006 gravatar image

When I scp the contents of _base from the cloud contoller to the nodes my instances also run fine and without problems (as far as I can see).

Regards John

edit flag offensive delete link more
0

answered 2011-03-17 16:22:48 -0500

jo0006 gravatar image

I am seeing the same problem. I have two nodes, one running api, network and compute and one just a compute node. Both nodes are running ubuntu 10.10, and the nova packages are installed from ppa trunk:

ii nova-api 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - API frontend ii nova-common 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - common files ii nova-compute 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - compute node ii nova-doc 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - documetation ii nova-network 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Network thingamajig ii nova-objectstore 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - object store ii nova-scheduler 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Scheduler ii python-nova 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Python libraries ii python-novaclient 2.4-0ubuntu1~lucid1 client library for OpenStack Compute API

When i launch an instance, if scheduled on the node running all the services (network, objectstore, scheduler) the instances boots and runs fine. On the compute only node:

2011-03-17 15:40:48,164 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000028/ from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,174 INFO nova.virt.libvirt_conn [-] instance instance-00000028: Creating image 2011-03-17 15:40:48,233 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/7faf7e4d /var/lib/nova/instances/instance-00000028/kernel from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,265 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/1b9c0a16 /var/lib/nova/instances/instance-00000028/disk from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,315 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/local_20 /var/lib/nova/instances/instance-00000028/disk.local from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,424 INFO nova.virt.libvirt_conn [-] instance instance-00000028: injecting key into image 463211030 2011-03-17 15:40:48,424 INFO nova.virt.libvirt_conn [-] instance instance-00000028: injecting net into image 463211030 2011-03-17 15:40:48,434 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-00000028/disk from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,460 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/nbd15 from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,474 DEBUG nova.utils [-] Result was 1 from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:161 2011-03-17 15:40:48,475 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -d /dev/nbd15 from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:40:48,490 WARNING nova.virt.libvirt_conn [-] instance instance-00000028: ignoring error injecting data ... (more)

edit flag offensive delete link more
0

answered 2011-03-18 00:36:00 -0500

I am seeing something like this, and it's go to do with the 2nd node installe via packages, with my investigations it looks like the instance is not being copied over to the node. My node was missing the _base directory under /var/lib/nova/instances yet the instance itself is created under this directory. Any help would be great!

here is my compute error log

011-03-17 15:29:42,987 INFO nova [-] called setup_basic_filtering in nwfilter 2011-03-17 15:29:42,987 INFO nova [-] ensuring static filters 2011-03-17 15:29:43,136 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x38ead10=""> 2011-03-17 15:29:43,136 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3bf1ad0=""> 2011-03-17 15:29:43,136 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3bf1950=""> 2011-03-17 15:29:43,137 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3bf1a50=""> 2011-03-17 15:29:43,145 DEBUG nova.utils [-] Attempting to grab iptables for method apply... from (pid=4027) inner /usr/lib/pymodules/python2.6/nova/utils.py:535 2011-03-17 15:29:43,153 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:29:43,163 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:29:43,173 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:29:43,182 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:29:43,201 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000015/ from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148 2011-03-17 15:29:43,206 INFO nova.virt.libvirt_conn [-] instance instance-00000015: Creating image 2011-03-17 15:29:43,277 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 424, in spawn (nova.exception): TRACE: self._create_image(instance, xml) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 609, in _create_image (nova.exception): TRACE: fname = '%08x' % int(disk_images['kernel_id']) (nova.exception): TRACE: ValueError: invalid literal for int() with base 10: 'ami-zmpaoemm' (nova.exception): TRACE: 2011-03-17 15:29:43,277 ERROR nova.compute.manager [X-XU2QSS54CYE169SRCB project admin-project] instance 21: Failed to spawn (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 218, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126 ... (more)

edit flag offensive delete link more
0

answered 2011-03-18 13:21:07 -0500

alberto-azns gravatar image

I have the same problem. I tried to manually copy the files inside _base directory from cloud controller to compute host and the instance go in running mode, but now I have a problem with the lan. I don't think the 2 problems are correlated. Bye, Alberto

edit flag offensive delete link more
0

answered 2011-03-20 07:34:06 -0500

alberto-azns gravatar image

I read http://wiki.openstack.org/LiveMigrationUsage (http://wiki.openstack.org/LiveMigrati...) about live migration between multiple nova servers: in the first part of article they make a NFS exprot of instance directory. At this point I don't think that _base directory will be copied automatically from hosts...

edit flag offensive delete link more
0

answered 2011-03-21 02:02:14 -0500

cuonghoangce gravatar image

Thanks John O'loughlin, that solved my question.

edit flag offensive delete link more
0

answered 2011-03-21 02:27:12 -0500

vishvananda gravatar image

Looks like there is some issue with the image service you are using.

The suggestion on this bug might be helpful:

https://bugs.launchpad.net/nova/+bug/...

You need to use GlanceImageService for multiple nodes. You also may need to convert your existing images from the old format using nova-manage image convert /path/to/image/dir

On Mar 20, 2011, at 7:02 PM, KiRaDo wrote:

Question #149424 on OpenStack Compute (nova) changed: https://answers.launchpad.net/nova/+q...

Status: Answered => Solved

KiRaDo confirmed that the question is solved: Thanks John O'loughlin, that solved my question.


You received this question notification because you are a member of Nova Core, which is an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-03-21 18:27:50 -0500

jo0006 gravatar image

Ok thanks, I'll look at the glance service.

Previously, as per the current documentation, I had nova-objectstore installed on the 'cloud controller' and then used the --s3_host in nova.conf to point the compute nodes at it.

So the recommendation is to install glance and run this in place of nova-objectstore ?

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-03-17 10:59:45 -0500

Seen: 64 times

Last updated: Mar 21 '11