Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Multi node deploy: instance spawning failure

Hello,

I have deployed latest nova on 2 host machines, one for controller(nova-api, nova-scheduler, nova-network, nova-objectstore) and the other for compute(nova-compute, nova-scheduler).

Problem is, as I hit run instance from the controller node, following compute error occurs on the compute node. nova-compute, nova-scheduler service was successfully added on the service table and euca-describe-instance show the state of instance spawning fail.

root@cloud02:~/openstack# nova-manage service list cloud02 nova-scheduler enabled :-) 2011-04-05 09:01:01 cloud02 nova-network enabled :-) 2011-04-05 09:01:01 cloud01 nova-scheduler enabled :-) 2011-04-05 08:59:23 cloud01 nova-compute enabled :-) 2011-04-05 08:59:17

root@cloud02:~/openstack# euca-describe-instances RESERVATION r-etss4atr NOVAADMPRJ default INSTANCE i-00000002 ami-0640184b 10.0.0.3 10.0.0.3 failed to spawn novaadm (NOVAADMPRJ, cloud01) 0 m1.small 2011-04-05T08:27:10Z nova

=============================================================================

2011-04-05 17:48:41,923 AUDIT nova.compute.manager [M39CSSE2RGAXIFEYZ6HY novaadm NOVAADMPRJ] instance 2: starting... 2011-04-05 17:48:42,018 DEBUG nova.rpc [-] Making asynchronous call on network.cloud02 ... from (pid=4290) call /root/openstack/nova/nova/rpc.py:345 2011-04-05 17:48:42,019 DEBUG nova.rpc [-] MSG_ID is 1d0b70e00fc94c14a7ed0bc91a17dd60 from (pid=4290) call /root/openstack/nova/nova/rpc.py:348 2011-04-05 17:48:42,458 DEBUG nova.virt.libvirt_conn [-] instance instance-00000002: starting toXML method from (pid=4290) to_xml /root/openstack/nova/nova/virt/libvirt_conn.py:899 2011-04-05 17:48:42,531 DEBUG nova.virt.libvirt_conn [-] instance instance-00000002: finished toXML method from (pid=4290) to_xml /root/openstack/nova/nova/virt/libvirt_conn.py:941 2011-04-05 17:48:42,582 INFO nova [-] called setup_basic_filtering in nwfilter 2011-04-05 17:48:42,582 INFO nova [-] ensuring static filters 2011-04-05 17:48:42,648 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=4290) inner /root/openstack/nova/nova/utils.py:594 2011-04-05 17:48:42,648 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=4290) inner /root/openstack/nova/nova/utils.py:599 2011-04-05 17:48:42,653 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=4290) execute /root/openstack/nova/nova/utils.py:150 2011-04-05 17:48:42,665 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=4290) execute /root/openstack/nova/nova/utils.py:150 2011-04-05 17:48:42,677 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=4290) execute /root/openstack/nova/nova/utils.py:150 2011-04-05 17:48:42,689 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=4290) execute /root/openstack/nova/nova/utils.py:150 2011-04-05 17:48:42,718 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /root/openstack/state/instances/instance-00000002/ from (pid=4290) execute /root/openstack/nova/nova/utils.py:150 2011-04-05 17:48:42,726 INFO nova.virt.libvirt_conn [-] instance instance-00000002: Creating image 2011-04-05 17:48:42,776 DEBUG nova.utils [-] Attempting to grab semaphore "115ec19b" for method "call_if_not_exists"... from (pid=4290) inner /root/openstack/nova/nova/utils.py:594 2011-04-05 17:48:42,777 DEBUG nova.utils [-] Running cmd (subprocess): cp /root/openstack/state/instances/_base/115ec19b /root/openstack/state/instances/instance-00000002/kernel from (pid=4290) execute /root/openstack/nova/nova/utils.py:150 2011-04-05 17:48:42,802 DEBUG nova.utils [-] Attempting to grab semaphore "0640184b" for method "call_if_not_exists"... from (pid=4290) inner /root/openstack/nova/nova/utils.py:594 2011-04-05 17:48:42,811 ERROR nova.compute.manager [M39CSSE2RGAXIFEYZ6HY novaadm NOVAADMPRJ] Instance '2' failed to spawn. Is virtualization enabled in the BIOS? (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/compute/manager.py", line 234, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/exception.py", line 120, in _wrap (nova.compute.manager): TRACE: return f(args, *kw) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/virt/libvirt_conn.py", line 563, in spawn (nova.compute.manager): TRACE: self._create_image(instance, xml, network_info) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/virt/libvirt_conn.py", line 783, in _create_image (nova.compute.manager): TRACE: size=size) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/virt/libvirt_conn.py", line 699, in _cache_image (nova.compute.manager): TRACE: call_if_not_exists(base, fn, args, *kwargs) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/utils.py", line 607, in inner (nova.compute.manager): TRACE: retval = f(args, *kwargs) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/virt/libvirt_conn.py", line 697, in call_if_not_exists (nova.compute.manager): TRACE: fn(target=base, args, *kwargs) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/virt/libvirt_conn.py", line 710, in _fetch_image (nova.compute.manager): TRACE: images.fetch(image_id, target, user, project) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/virt/images.py", line 51, in fetch (nova.compute.manager): TRACE: metadata = image_service.get(elevated, image_id, image_file) (nova.compute.manager): TRACE: File "/root/openstack/nova/nova/image/local.py", line 113, in get (nova.compute.manager): TRACE: raise exception.NotFound (nova.compute.manager): TRACE: NotFound: None (nova.compute.manager): TRACE: libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000002' 2011-04-05 17:48:55,846 INFO nova.compute.manager [-] Found instance 'instance-00000002' in DB but no VM. State=8, so setting state to shutoff. 2011-04-05 17:48:55,847 INFO nova.compute.manager [-] DB/VM state mismatch. Changing state from '8' to '5'