Ask Your Question
0

Adding a second compute node fails

asked 2011-04-15 22:39:47 -0600

patryk-kaminski gravatar image

I'm trying to install OpenStack on two host computers. The first one (192.168.64.66) is for the controller (nova-api, nova-scheduler, nova-objectstore, nova-network) and compute (nova-compute). The second host computer should run only the compute (nova-compute and anything else that is required... nova-schedule?). I use flat networking model. Both host computers run Ubuntu 10.04.02 x64.

I have installed the nova packages from ppa:nova-core/release, and then used the the https://github.com/dubsquared/OpenStack-NOVA-Installer-Script/raw/master/nova-CC-install-v1.1.sh (https://github.com/dubsquared/OpenSta...) script to configure the software on the first host computer. I was able to start instances, SSH to them and access the network from within the instances. No issues here.

Next I installed the nova packages on the second host (also from ppa:nova-core/release) and the used the https://github.com/dubsquared/OpenStack-NOVA-Installer-Script/raw/master/nova-NODE-installer.sh (https://github.com/dubsquared/OpenSta...) script to configure the software.

After restarting the nova services on both computers, nova-manage service list reports the following: asa-test-a nova-network enabled :-) 2011-04-15 01:18:00 asa-test-a nova-compute enabled :-) 2011-04-15 01:18:00 asa-test-a nova-scheduler enabled :-) 2011-04-15 01:18:00 asa-test-b nova-scheduler enabled XXX 2011-04-14 23:16:15

Clearly there is an issue with the nova-scheduler on the second host (asa-test-b). Do I need it at all since I already have the nova-scheduler running on the first host (asa-test-a)? I do not see the nova-compute running on the second host. (log output later, below). Should nova-network be running on the second host? It seems to fail (log output later, below).

/etc/network/interfaces file from the first host (asa-test-a):

auto lo iface lo inet loopback

auto br100 iface br100 inet static bridge_ports eth0 bridge_stp off bridge_maxwait 0 bridge_fd 0 address 192.168.64.66 netmask 255.255.255.0 broadcast 192.168.64.255 gateway 192.168.64.1 dns-nameservers 192.168.64.1

/etc/network/interfaces file from the second host (asa-test-b):

auto lo iface lo inet loopback

auto br100 iface br100 inet static bridge_ports eth0 bridge_stp off bridge_maxwait 0 bridge_fd 0 address 192.168.64.67 netmask 255.255.255.0 broadcast 192.168.64.255 gateway 192.168.64.1 dns-nameservers 192.168.64.1

/etc/nova/nova.conf (identical on both hosts):

--dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova --s3_host=192.168.64.66 --rabbit_host=192.168.64.66 --cc_host=192.168.64.66 --ec2_url=http://192.168.64.66:8773/services/Cloud --fixed_range=192.168.64.0/24 --network_size=64 --FAKE_subdomain=ec2 --routing_source_ip=192.168.64.66 --verbose --sql_connection=mysql://root:nova@192.168.64.66/nova --network_manager=nova.network.manager.FlatManager

The last few entries from the /var/log/nova/nova-compute.log file on asa-test-b:

(nova.root): TRACE: Traceback (most recent call last): (nova.root): TRACE: File "/usr/bin/nova-compute", line 44, in <module> (nova.root): TRACE: service.serve() (nova.root): TRACE: File "/usr/lib/pymodules/python2.6 ...

(more)
edit retag flag offensive close merge delete
add a comment

4 answers

Sort by ยป oldest newest most voted
0

answered 2011-04-17 02:11:08 -0600

dan-prince gravatar image

Hi Patryk,

From the looks of the stack trace it appears that you are running a nova database schema for cactus (or very close to it) with an older nova-compute client.

There were some changes in the Cactus dev cycle to the instances table. The instances.instance_type column has been replaces with instances.instance_type_id.

Can you verify the PPA package version on your machines by running a dpkg -l and looking at the revision?

Hope this helps.

Dan

edit flag offensive delete link more
add a comment
0

answered 2011-04-18 15:11:28 -0600

patryk-kaminski gravatar image

Hi Dan,

Thank you for your reply. You were right. I checked the PPA package versions, and found a mismatch. I then reinstalled the nova packages. Running dpkg -l | grep nova on either host server returns:

ii nova-api 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - API frontend ii nova-common 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - common files ii nova-compute 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - compute node ii nova-doc 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - documetation ii nova-network 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Network thingamaj ii nova-objectstore 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - object store ii nova-scheduler 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Scheduler ii python-nova 2011.2-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Python libraries ii python-novaclient 2.4-0ubuntu1~lucid1 client library for OpenStack Compute API

After restarting both host servers, I get the following output from nova-manage service list: asa-test-a nova-compute enabled :-) 2011-04-17 17:01:59 asa-test-a nova-network enabled :-) 2011-04-17 17:01:57 asa-test-a nova-scheduler enabled :-) 2011-04-17 17:01:57 asa-test-b nova-scheduler enabled XXX 2011-04-17 17:01:06 asa-test-b nova-network enabled XXX 2011-04-17 17:01:05 asa-test-b nova-compute enabled XXX 2011-04-17 17:01:11

I tried starting an instance, ex.: euca-run-instances $emi -k mykey -t m1.tiny I can see that it now tries to launch the instance (sometimes on the primary host, sometimes on the secondary host). They run fine on first host (asa-test-a) but on the second host (asa-test-b) the instance status quickly changes from scheduling to launching to shutdown.
Despite the suggestion in the nova-compute.log, both machines have support for hardware virtualization, and I have successfully run kvm on both of them.

Here is the output from the nova-compute.log file on second host server (asa-test-b) when it tires to launch an instance:

2011-04-17 12:55:08,260 AUDIT nova.compute.manager [Q6TB2JI-BQUULBHOXPGI admin asa] instance 16: starting... 2011-04-17 12:55:08,959 INFO nova [-] called setup_basic_filtering in nwfilter 2011-04-17 12:55:08,960 INFO nova [-] ensuring static filters 2011-04-17 12:55:09,146 INFO nova.virt.libvirt_conn [-] instance instance-00000010: Creating image 2011-04-17 12:55:09,412 INFO nova.virt.libvirt_conn [-] instance instance-00000010: injecting key into image 1853417834 2011-04-17 12:55:09,412 INFO nova.virt.libvirt_conn [-] instance instance-00000010: injecting net into image 1853417834 2011-04-17 12:55:10,512 WARNING nova.virt.libvirt_conn [-] instance instance-00000010: ignoring error injecting data into image 1853417834 (Unexpected error while running command. Command: sudo tune2fs -c 0 -i 0 /dev/nbd15 Exit code: 1 Stdout: 'tune2fs 1.41.11 (14-Mar-2010)\n' Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\r\nCouldn't find valid filesystem superblock.\n") 2011-04-17 12:55:41,903 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 617, in spawn (nova.exception): TRACE: domain = self._create_new_domain(xml) (nova.exception): TRACE: File "/usr/lib/pymodules ...

(more)
edit flag offensive delete link more
add a comment
0

answered 2011-04-18 20:17:19 -0600

patryk-kaminski gravatar image

Well, never mind. Looking at other posts on this forum, I realized that I cannot use local storage for multiple hosts. I have installed glance and now I can start instances on either host server.

P.

edit flag offensive delete link more
add a comment
0

answered 2011-04-18 20:18:21 -0600

patryk-kaminski gravatar image

Thanks Dan Prince, that solved my question.

edit flag offensive delete link more
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2011-04-15 22:39:47 -0600

Seen: 125 times

Last updated: Apr 18 '11

Related questions

euca-describe-images does not work

Nova Class not found error

Can not create keypair

euca-allocate-address failed

What do I need to get started?

Installing Nova on Ubuntu 10.04

XenServer installation

Ephemeral/Persistent disks

Direct EC2 support?

how to delete the volume ?