Ask Your Question
0

Startup problems with a compute node in a multi-node cluster

asked 2011-06-28 21:21:29 -0500

cubranic gravatar image

I am setting up a dual-node cluster, with one node running all services (let's call it nova1), and another just nova-compute (nova2). The first node works fine, but on the latter, the compute node nova2, nova-compute service does not start properly. In the syslog, I see lines like:

Jun 28 06:25:14 compute1 init: nova-compute main process (2914) terminated with status 1 Jun 28 06:25:14 compute1 init: nova-compute main process ended, respawning

(repeating every second)

In /var/log/nova/nova-compute.log, I see the following CRITICAL error with a stack trace: (OperationalError) (1054, "Unknown column 'instances.image_ref' in 'field list'")"

Probably because of this error, Nova never sets up the networking bridges and routes, and the compute node cannot access guest instances running on the nova1 node.

Interestingly, the command line tools that I tried on nova2 ("euca-describe-instances", "nova-manage network list", etc.) still show correct information about the Nova cluster and instances running on nova1.

edit retag flag offensive close merge delete

21 answers

Sort by » oldest newest most voted
0

answered 2011-07-04 19:58:32 -0500

I added this bug to the openstack-manuals project:

https://bugs.launchpad.net/openstack-manuals/+bug/804099 (https://bugs.launchpad.net/openstack-...)

I think that is all we need for now. Please file any other discrepancies you find in the docs in that project as well. Thanks!

edit flag offensive delete link more
0

answered 2011-07-04 19:55:28 -0500

cubranic gravatar image

Thanks Brian Waldon, that solved my question.

edit flag offensive delete link more
0

answered 2011-07-04 19:54:56 -0500

cubranic gravatar image

Thanks for the explanation Brian. I think part of the problem is that the Cactus docs use "nova-trunk" PPA, so anyone following them now will run into problems because they won't know that Glance needs to be available. I'll add a comment to the page that references the trunk, but please let me know if it would also be useful to open a Launchpad bug for this.

On a second look, the automated install script might not have this problem because it uses the "release" ppa, so people using that will have no problems.

edit flag offensive delete link more
0

answered 2011-06-30 21:38:44 -0500

Thank you for your help, Davor. I will make sure our image service documentation is up to date. Keep in mind the docs you refer to are for our latest official release, Cactus. If you are using the Diablo trunk or milestone packages, it isn't safe to refer to those docs.

I think the question here has been answered, please reopen it if you feel otherwise.

edit flag offensive delete link more
0

answered 2011-06-30 21:22:14 -0500

cubranic gravatar image

Really? You guys know that there is no mention of this in the official (?) docs on docs.openstack.org? And looking at the install script that you provide [1], there is no handling of glance at all in its code, so I assume that it also expects to use the LocalImageService.

[1] https://raw.github.com/elasticdog/OpenStack-NOVA-Installer-Script/master/nova-install (https://raw.github.com/elasticdog/Ope...) )

edit flag offensive delete link more
0

answered 2011-06-30 12:54:10 -0500

You may have installed Nova before Glance was required. We removed the LocalImageService recently in favor of the filesystem backend in Glance. Right now, your installation seems to be looking at the default localhost:9292, that's why you're seeing the connection errors. You'll need to install Glance and configure the 'glance_api_servers' flag in your Nova config. See http://glance.openstack.org for installation help.

edit flag offensive delete link more
0

answered 2011-06-30 00:07:03 -0500

cubranic gravatar image

I don't have a running glance server, and one is not configured in nova.conf. Is this a new requirement? I didn't seem to need it when I first installed OpenStack a few weeks ago (following the steps for manual install in http://docs.openstack.com ): euca-describe-images worked until I updated Nova packages yesterday.

edit flag offensive delete link more
0

answered 2011-06-29 17:28:59 -0500

blamar gravatar image

Davor,

"TypeError: 'NoneType' object does not support item assignment"

This is a bug that was recently fixed in Glance: https://bugs.launchpad.net/glance/+bug/803188 (https://bugs.launchpad.net/glance/+bu...)

edit flag offensive delete link more
0

answered 2011-06-29 17:03:42 -0500

cubranic gravatar image

Also, still no Nova-related networking is set up on the second node (br100 and routes/iptables rules to ping the VMs).

edit flag offensive delete link more
0

answered 2011-06-29 17:01:04 -0500

cubranic gravatar image

Thanks Vish, running "nova-manage db sync" got me to the point where I can again run and access instances on node1. But I still see the following error in nova-api.log when I run "euca-describe-images":

Unexpected error raised: 'NoneType' object does not support item assignment (nova.api): TRACE: Traceback (most recent call last): (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/api/ec2/__init__.py ", line 320, in __call__ (nova.api): TRACE: result = api_request.invoke(context) (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/api/ec2/apirequest. py", line 78, in invoke (nova.api): TRACE: result = method(context, **args) (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/api/ec2/cloud.py", line 1097, in describe_images (nova.api): TRACE: images = self.image_service.detail(context) (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/image/s3.py", line 75, in detail (nova.api): TRACE: return self.service.detail(context) (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/image/glance.py", line 106, in detail (nova.api): TRACE: limit=limit) (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/glance/client.py", line 85, in get_images_detailed (nova.api): TRACE: params = self._extract_params(kwargs, v1_images.SUPPORTED_PARAMS) (nova.api): TRACE: File "/usr/lib/pymodules/python2.7/glance/common/client.py", line 174, in _extract_params (nova.api): TRACE: result[allowed_param] = actual_params[allowed_param] (nova.api): TRACE: TypeError: 'NoneType' object does not support item assignment (nova.api): TRACE:

Restarting nova-api service does not help, the error is still there.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-06-28 21:21:29 -0500

Seen: 195 times

Last updated: Jul 04 '11