When creating an instance from a specific image, I get a failure.

asked 2014-05-29 05:25:24 -0600

Avi Zohary gravatar image

updated 2014-05-31 23:55:28 -0600

I have a certain image (Oracle Linux with some applications on it) that I've been creating instances from for the past few weeks. A couple of days ago, creating instances from that images began failing repeatedly, while creating instances from another image does seem to work (the main difference between the two images, as far as I can tell is that the failing one comes with cloud-init installed, and the working one doesn't).

At the moment I'm testing this with the Horizon UI, simply trying to create an instance.

I looked at the nova compute.log on the compute node and I see an exception (pasted at the end) basically saying

2014-05-29 13:05:41.865 29768 ERROR nova.compute.manager [req-edd9d436-b48a-49ee-9dbd-fe40d7bef362 2bf9c3b4b60b4423bc70c5a4d8a18b3c 302b6eee9f9d49d18c4ddb9caea4daf5] [instance: da98e52e-d8c1-43a0-b4c3-533e88b7f95d] Instance failed to spawn

I'm not sure what is the cause for this - first I'm not sure what might cause this, and second, what changed - I know that the image hasn't changed, and I don't remember making any modifications to the configuration files that may have caused this.

Is there a good way to analyze this issue - some way to access boot logs, see what happened while the instance tried to spawn and failed?

I tried opening debug logs, but couldn't find something that looks meaningful W.R.T. why this failure occurs.

Here is some more information about my installation:

Using Openstack Havana (glance, keystone, horizon, nova, heat and ceilometer), split between controller and compute node. The controller node doesn't have the nova-compute service enabled, so all instances are created on the compute node.

Regarding the images - I have a few images, all in qcow2 format (created them in virtual box). All but two of them seem to be working fine - I can create instances from them, and the other two, basically the same image, are the ones I'm having problems with.

The image contents in all cases is Oracle Linux (Basically RHEL), with a Weblogic application installed on them. The images that are working are those without cloud-init on them, and the ones that don't are the ones without.

I'd appreciate any help!

Thanks. Avi

Libvirt instance log:

2014-05-29 11:46:32.956+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-000000e7 -S -M rhel6.5.0 -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid a487b3e2-4ae2-47e1-b596-0e919b049399 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.3-1.el6,serial=e00d6db6-0010-ffff-ffff-ffffff200008,uuid=a487b3e2-4ae2-47e1-b596-0e919b049399 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000000e7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/nova/instances/a487b3e2-4ae2-47e1-b596-0e919b049399/disk ...
(more)
edit retag flag offensive close merge delete

Comments

Hi, Can you please paste some more lines(as many as possible) of compute.log before "Instance failed to spawn" ?

Itzik gravatar imageItzik ( 2014-05-29 14:47:18 -0600 )edit

Actually, I restarted the nova-compute process on the compute node, and cleared the log before creating the instance, so what I pasted above is the full log that I have. I'll redo this, and let the log run for a while before creating the instance.

Thanks avi

Avi Zohary gravatar imageAvi Zohary ( 2014-05-31 23:38:35 -0600 )edit

Added the second log at the end of the question.

Thanks. Avi

Avi Zohary gravatar imageAvi Zohary ( 2014-05-31 23:56:04 -0600 )edit

Can you check SELinux ? https://bugzilla.redhat.com/show_bug.cgi?id=896706 (https://bugzilla.redhat.com/show_bug....)

Itzik gravatar imageItzik ( 2014-06-01 04:28:41 -0600 )edit

Hey Itzik

I tried setting selinux to be permissive, but still got the same results.

BTW - I ruled out any relation to cloud-init - got the same result on a new image I created from which I removed the cloud-init.

Thanks. Avi

Avi Zohary gravatar imageAvi Zohary ( 2014-06-01 14:38:52 -0600 )edit