multi node instance failure

asked 2011-03-30 23:06:22 -0600

jon-whiteheat
1 ●1 ●1 ●1

Hi,

I've 6 machines I want to run OpenStack on. I'm planning on one controller, and the rest nodes.

I've configured the controller to run instances locally, using ubuntu1010-UEC-localuser-image.tar.gz, and I'm able to ping and SSH to them, etc. Yay :D The joy.

I then add a compute node. I'm manually installing parts to get a feel for what's happening. Here's what I do:-

$ sudo apt-get install python-software-properties $ sudo add-apt-repository ppa:nova-core/trunk $ sudo apt-get update $ sudo apt-get install -y nova-common nova-compute python-mysqldb python-suds vlan

I'll then scp nova.conf from the controller to the node, and restart all services on all machines, after nova-manage db sync on the controller. I don't edit network/interfaces for br100 as I'm running VlanManager, this could be a mistake. The node will then appear in the services table.

When I attempt to run instances, and when they're scheduled for on the additional nodes the instances always fail. Here are 2 log files, from 2 nodes I've tried to configure:-

CLOUD NODE 1

2011-03-30 17:07:34,326 WARNING nova.virt.libvirt_conn [-] instance instance-00000014: ignoring error injecting data into image 808928202 (Unexpected error whi le running command. Command: sudo tune2fs -c 0 -i 0 /dev/nbd15 Exit code: 1 Stdout: 'tune2fs 1.41.12 (17-May-2010)\n' Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n") 2011-03-30 17:08:05,516 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn (nova.exception): TRACE: self._conn.createXML(xml, 0) (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) (nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1 (nova.exception): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ioctl for device (nova.exception): TRACE: (nova.exception): TRACE: 2011-03-30 17:08:05,518 ERROR nova.compute.manager [E44FOOY3XA5BTH21IXRG admin cloud] Instance '20' failed to spawn. Is virtualization enabled in the B IOS? (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap (nova.compute.manager): TRACE: raise Error(str(e)) (nova.compute.manager): TRACE: Error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1 (nova.compute.manager): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ... (more)

edit retag flag offensive close merge delete

add a comment

9 answers

Sort by » oldest newest most voted

answered 2011-03-30 23:20:11 -0600

vishvananda
366 ●2 ●2 ●3

There are probably a couple of issues here. One is that you will need to use glance so that that the other node can grab images. You also may have issues with kvm module and permissions:

sudo modprobe kvm sudo chgroup kvm /dev/kvm sudo chmod g+w /dev/kvm

Vish

On Mar 30, 2011, at 4:06 PM, nimda wrote:

New question #151125 on OpenStack Compute (nova): https://answers.launchpad.net/nova/+q...

Hi,

I've 6 machines I want to run OpenStack on. I'm planning on one controller, and the rest nodes.

I've configured the controller to run instances locally, using ubuntu1010-UEC-localuser-image.tar.gz, and I'm able to ping and SSH to them, etc. Yay :D The joy.

I then add a compute node. I'm manually installing parts to get a feel for what's happening. Here's what I do:-

$ sudo apt-get install python-software-properties $ sudo add-apt-repository ppa:nova-core/trunk $ sudo apt-get update $ sudo apt-get install -y nova-common nova-compute python-mysqldb python-suds vlan

I'll then scp nova.conf from the controller to the node, and restart all services on all machines, after nova-manage db sync on the controller. I don't edit network/interfaces for br100 as I'm running VlanManager, this could be a mistake. The node will then appear in the services table.

When I attempt to run instances, and when they're scheduled for on the additional nodes the instances always fail. Here are 2 log files, from 2 nodes I've tried to configure:-

CLOUD NODE 1

2011-03-30 17:07:34,326 WARNING nova.virt.libvirt_conn [-] instance instance-00000014: ignoring error injecting data into image 808928202 (Unexpected error whi le running command. Command: sudo tune2fs -c 0 -i 0 /dev/nbd15 Exit code: 1 Stdout: 'tune2fs 1.41.12 (17-May-2010)\n' Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n") 2011-03-30 17:08:05,516 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn (nova.exception): TRACE: self._conn.createXML(xml, 0) (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) (nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1 (nova.exception): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ioctl for device (nova.exception): TRACE: (nova.exception): TRACE: 2011-03-30 17:08:05,518 ERROR nova.compute.manager [E44FOOY3XA5BTH21IXRG admin cloud] Instance '20' failed to spawn. Is virtualization enabled in the B IOS? (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager ...

(more)

edit flag offensive delete link

add a comment

answered 2011-03-31 16:46:57 -0600

jon-whiteheat
1 ●1 ●1 ●1

I've tested a qemu image on both nodes, and was able to execute a VM. I'm still receiving exactly the same errors in nova-compute log when trying to launch OpenStack instances though.

I'll look into setting up Glance, to see what effect that has on my experience. Are there any guides to set up Glance?

edit flag offensive delete link

add a comment

answered 2011-03-31 16:58:58 -0600

jon-whiteheat
1 ●1 ●1 ●1

I tried to install an additional 2 node, just to see what sort of results I got. I'm starting to see a pattern now:-

NODE 3

2011-03-31 17:56:35,236 WARNING nova.virt.libvirt_conn [-] instance instance-0000001e: ignoring error injecting data into image 808928202 (Unexpected error whi le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-31 17:56:37,437 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(args, *kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-31 17:56:37,439 ERROR nova.compute.manager [7QVL-ILR6YXMO5S80MY9 positive posi-cloud] Instance '30' failed to spawn. Is virtualization enabled in the B IOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.compute.manager): TRACE:

NODE 4

2011-03-31 17:56:38,218 WARNING nova.virt.libvirt_conn [-] instance instance-00000021: ignoring error injecting data into image 808928202 (Unexpected error whi le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-31 17:56:40,447 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(args, *kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-31 17:56:40,449 ERROR nova.compute.manager [WMV3POEJTG1QY89TH31R positive posi-cloud] Instance '33' failed to spawn. Is virtualization enabled ...

(more)

edit flag offensive delete link

add a comment

answered 2011-04-06 19:47:51 -0600

jon-whiteheat
1 ●1 ●1 ●1

Okay, I've had some success. I'm able to create instances on all nodes, only I'm not able to ping them. I was able to do this by copying /var/lib/nova/instances/_base/ from the controller to the node.

edit flag offensive delete link

add a comment

answered 2011-04-06 19:49:54 -0600

jon-whiteheat
1 ●1 ●1 ●1

Additionally, I'm not able to ping any instance from the 2nd node.

edit flag offensive delete link

add a comment

answered 2011-03-31 17:02:18 -0600

jaypipes

281 ●1 ●3 ●8 http://joinfu.com/

On Thu, Mar 31, 2011 at 12:46 PM, nimda question151125@answers.launchpad.net wrote:

I'll look into setting up Glance, to see what effect that has on my experience. Are there any guides to set up Glance?

Sure thing.

http://glance.openstack.org/installin... http://glance.openstack.org/controlli...

Working on the configuration documentation...

Cheers, jay

edit flag offensive delete link

add a comment

answered 2011-04-04 16:14:55 -0600

jon-whiteheat
1 ●1 ●1 ●1

I'd like to know if creating instances across multiple nodes requires setting up, past what's told in this guide:-

http://docs.openstack.org/openstack-compute/admin/content/ch01.html (http://docs.openstack.org/openstack-c...)

I can create a single node, that's able to create instances, as mentioned above, but not onto additional nodes. Is the guide above missing anything I need to know to accomplish this?

I had started looking into Glance, as that's what similar questions have been resolved with. But, since reading about Glance, I'm now understanding I should be able to use nova's own object-store.

At the very least, is there a simple guide to create a multi-node cloud, that I can learn from?

edit flag offensive delete link

add a comment

answered 2011-04-06 22:00:39 -0600

jon-whiteheat
1 ●1 ●1 ●1

This evening I managed to get instances on on controller/node and a standalone node, and was able to communicate with them.

I've tweaked many different parts so I'm not entirely sure what combination finally worked. I have tried to install another node, and that failed, I think this is because of missing /var/lib/nova/instances/_base/ files. I'll look into this.

So here's nova.conf

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--sql_connection=mysql://root:88bombs4@192.168.99.11/nova --s3_host=192.168.99.11
--rabbit_host=192.168.99.11
--cc_host=192.168.99.11
--ec2_url=http://192.168.99.11:8773/services/Cloud
--network_manager=nova.network.manager.FlatDHCPManager
--flat_interface=eth0
--fixed_range=10.0.0.0/12
--network_size=32
--quota_instances=100

...and /etc/network/interfaces

The loopback network interface

auto lo
iface lo inet loopback

The primary network interface

auto eth0
iface eth0 inet dhcp

Other than that, I just followed this guide http://www.dubsquared.com . I can't yet connect to the instances publicly so that'll be my next task.

edit flag offensive delete link

add a comment

answered 2011-07-21 07:13:42 -0600

sunny790821
1 ●1 ●1 ●1

I got similar message. T.T I use XCP(xenServer community version) as hypervisor. Some has the answer?

2011-07-21 16:01:21,767 ERROR nova.compute.manager [X1HKIK61WUJ648ADHZZE novaadmin proj] Instance '77' failed to spawn. Is virtualization enabled in the BIO S? (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi_conn.py", line 188, in spawn (nova.compute.manager): TRACE: self._vmops.spawn(instance) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 117, in spawn (nova.compute.manager): TRACE: vdi_uuid = self._create_disk(instance) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 113, in _create_disk (nova.compute.manager): TRACE: instance.image_id, user, project, disk_image_type) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 386, in fetch_image (nova.compute.manager): TRACE: access, image_type) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 541, in _fetch_image_glance (nova.compute.manager): TRACE: session, instance_id, image, access, image_type) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 456, in _fetch_image_glance_disk (nova.compute.manager): TRACE: lambda dev: (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 944, in with_vdi_attached_here (nova.compute.manager): TRACE: return f(dev) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 458, in <lambda> (nova.compute.manager): TRACE: virtual_size, image_file)) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1013, in _stream_disk (nova.compute.manager): TRACE: _write_partition(virtual_size, dev) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1033, in _write_partition (nova.compute.manager): TRACE: execute('sudo', 'parted', '--script', dest, 'mklabel', 'msdos') (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1031, in execute (nova.compute.manager): TRACE: return utils.execute(cmd, *kwargs) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 173, in execute (nova.compute.manager): TRACE: cmd=' '.join(cmd)) (nova.compute.manager): TRACE: ProcessExecutionError: Unexpected error while running command. (nova.compute.manager): TRACE: Command: sudo parted --script /dev/xvdb mklabel msdos (nova.compute.manager): TRACE: Exit code: 1 (nova.compute.manager): TRACE: Stdout: '' (nova.compute.manager): TRACE: Stderr: 'sudo: no tty present and no askpass program specified\n' (nova.compute.manager): TRACE: (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi_conn.py", line 188, in ... (more)

edit flag offensive delete link

add a comment

multi node instance failure

9 answers

The loopback network interface

The primary network interface

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

multi node instance failure edit

9 answers

The loopback network interface

The primary network interface

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

multi node instance failure