Ask Your Question

Avi Zohary's profile - activity

2014-08-06 08:50:36 -0500 received badge  Famous Question (source)
2014-06-04 13:42:23 -0500 received badge  Necromancer (source)
2014-06-04 13:42:23 -0500 received badge  Teacher (source)
2014-06-02 02:06:35 -0500 received badge  Notable Question (source)
2014-06-01 14:38:52 -0500 commented question When creating an instance from a specific image, I get a failure.

Hey Itzik

I tried setting selinux to be permissive, but still got the same results.

BTW - I ruled out any relation to cloud-init - got the same result on a new image I created from which I removed the cloud-init.

Thanks. Avi

2014-05-31 23:56:04 -0500 commented question When creating an instance from a specific image, I get a failure.

Added the second log at the end of the question.

Thanks. Avi

2014-05-31 23:38:35 -0500 commented question When creating an instance from a specific image, I get a failure.

Actually, I restarted the nova-compute process on the compute node, and cleared the log before creating the instance, so what I pasted above is the full log that I have. I'll redo this, and let the log run for a while before creating the instance.

Thanks avi

2014-05-31 14:55:43 -0500 received badge  Popular Question (source)
2014-05-29 07:10:43 -0500 received badge  Editor (source)
2014-05-29 07:04:07 -0500 answered a question nova instance fails with libvirtError: internal error Process exited while reading console log output

Here is some more information about my installation:

Using Openstack Havana (glance, keystone, horizon, nova, heat and ceilometer), split between controller and compute node. The controller node doesn't have the nova-compute service enabled, so all instances are created on the compute node.

Regarding the images - I have a few images, all in qcow2 format (created them in virtual box). All but two of them seem to be working fine - I can create instances from them, and the other two, basically the same image, are the ones I'm having problems with.

The image contents in all cases is Oracle Linux (Basically RHEL), with a Weblogic application installed on them. The images that are working are those without cloud-init on them, and the ones that don't are the ones without.

BTW, here is a the instance log that was created in the /var/log/libvirt/qemu/instance-...log :

2014-05-29 11:46:32.956+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-000000e7 -S -M rhel6.5.0 -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid a487b3e2-4ae2-47e1-b596-0e919b049399 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.3-1.el6,serial=e00d6db6-0010-ffff-ffff-ffffff200008,uuid=a487b3e2-4ae2-47e1-b596-0e919b049399 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000000e7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/nova/instances/a487b3e2-4ae2-47e1-b596-0e919b049399/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c1:53:56,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/home/nova/instances/a487b3e2-4ae2-47e1-b596-0e919b049399/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:5 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/8
qemu-kvm: -drive file=/home/nova/instances/a487b3e2-4ae2-47e1-b596-0e919b049399/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /home/nova/instances/a487b3e2-4ae2-47e1-b596-0e919b049399/disk: Operation not supported
2014-05-29 11:46:33.338+0000: shutting down

And this is what I see in /var/log/libvirt/libvirtd.log:

2014-05-29 11:46:33.338+0000: 2730: error : qemuMonitorIORead:514 : Unable to read from monitor: Connection reset by peer
2014-05-29 11:46:33.338+0000: 2734: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet5" not in key map
2014-05-29 11:46:33.356+0000: 2734: error : virNetDevGetIndex:653 : Unable to get index for interface vnet5: No such device
2014-05-29 05:25:24 -0500 asked a question When creating an instance from a specific image, I get a failure.

I have a certain image (Oracle Linux with some applications on it) that I've been creating instances from for the past few weeks. A couple of days ago, creating instances from that images began failing repeatedly, while creating instances from another image does seem to work (the main difference between the two images, as far as I can tell is that the failing one comes with cloud-init installed, and the working one doesn't).

At the moment I'm testing this with the Horizon UI, simply trying to create an instance.

I looked at the nova compute.log on the compute node and I see an exception (pasted at the end) basically saying

2014-05-29 13:05:41.865 29768 ERROR nova.compute.manager [req-edd9d436-b48a-49ee-9dbd-fe40d7bef362 2bf9c3b4b60b4423bc70c5a4d8a18b3c 302b6eee9f9d49d18c4ddb9caea4daf5] [instance: da98e52e-d8c1-43a0-b4c3-533e88b7f95d] Instance failed to spawn

I'm not sure what is the cause for this - first I'm not sure what might cause this, and second, what changed - I know that the image hasn't changed, and I don't remember making any modifications to the configuration files that may have caused this.

Is there a good way to analyze this issue - some way to access boot logs, see what happened while the instance tried to spawn and failed?

I tried opening debug logs, but couldn't find something that looks meaningful W.R.T. why this failure occurs.

Here is some more information about my installation:

Using Openstack Havana (glance, keystone, horizon, nova, heat and ceilometer), split between controller and compute node. The controller node doesn't have the nova-compute service enabled, so all instances are created on the compute node.

Regarding the images - I have a few images, all in qcow2 format (created them in virtual box). All but two of them seem to be working fine - I can create instances from them, and the other two, basically the same image, are the ones I'm having problems with.

The image contents in all cases is Oracle Linux (Basically RHEL), with a Weblogic application installed on them. The images that are working are those without cloud-init on them, and the ones that don't are the ones without.

I'd appreciate any help!

Thanks. Avi

Libvirt instance log:

2014-05-29 11:46:32.956+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-000000e7 -S -M rhel6.5.0 -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid a487b3e2-4ae2-47e1-b596-0e919b049399 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.3-1.el6,serial=e00d6db6-0010-ffff-ffff-ffffff200008,uuid=a487b3e2-4ae2-47e1-b596-0e919b049399 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000000e7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/nova/instances/a487b3e2-4ae2-47e1-b596-0e919b049399/disk ...
(more)