Ask Your Question
0

Glance image delivery not working on compute nodes

asked 2011-07-09 14:34:21 -0500

howard-h gravatar image

Hi all,

Glance image delivery doesn't seem to be working for me!

I have a three server set-up with Glance running on the cloud controller which also runs compute. Instances that start on the cloud controller work fine but those that attempt to start on the compute nodes always fail. I have the same nova.conf on the nodes and controller. The relevant part of the compute log on the node is:

2011-07-09 15:07:52,811 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/nbd15 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-09 15:07:52,856 DEBUG nova.utils [-] Result was 1 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:166 2011-07-09 15:07:52,857 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -d /dev/nbd15 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-09 15:07:52,906 WARNING nova.virt.libvirt_conn [-] instance instance-00000018: ignoring error injecting data into image 2 (Unexpected error while running command. Command: sudo tune2fs -c 0 -i 0 /dev/nbd15 Exit code: 1 Stdout: 'tune2fs 1.41.14 (22-Dec-2010)\n' Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n") 2011-07-09 15:08:25,952 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last):

Additionally, the files in /var/lib/nova/instances/_base are zero length; I've seen that others have copied the contents of the controller's version of this directory into here but this seems to defeat the object of Glance.

Also, "glance index --host=10.0.0.100" on a node gives the following error (it works fine on the controller):

Failed to show index. Got error: Unknown error occurred! {"versions": [{"status": "CURRENT", "id": "v1.0", "links": [{"href": "http://0.0.0.0:9292/v1/", "rel": "self"}]}]} howard@stratus:~$ ping 192.168.1.44

Can anyone give me some suggestions of a solution or further diagnostic steps--others have posted the same problem but I've not seen a solution.

Thanks

edit retag flag offensive close merge delete

23 answers

Sort by ยป oldest newest most voted
0

answered 2011-08-11 20:33:25 -0500

howard-h gravatar image

I'll give that a go and post an update--I'm using Cactus.

edit flag offensive delete link more
0

answered 2011-08-11 18:00:33 -0500

forgot to mention that we're using "diablo d3" from the ppa:nova-core/milestone

edit flag offensive delete link more
0

answered 2011-08-11 03:30:01 -0500

we faced the same issue and after following Jay's advice, we issued the following commands to Compute node and then it worked! (so that now we no longer need to manually move /var/lib/nova/instances/_base from Controller node to Compute node to launch/run instances on Compute node.)

sudo apt-get purge nova-compute sudo apt-get install glance sudo purge nova-common sudo apt-get install nova-compute mkdir /var/lib/nova/instances chmod 777 /var/lib/nova/instances sudo /etc/init.d/libvirt-bin restart sudo restart nova-compute

(note: we didn't do sync_db on Compute node)

daniel

edit flag offensive delete link more
0

answered 2011-07-29 20:25:18 -0500

jaypipes gravatar image

Hi Howard,

I'm not sure what I can do here. There's been some sort of corruption of the registry database, where the db_sync command is bombing out. The corruption is due to the registry database apparently not having the correct database version record in the migration table. I'm not sure what I can recommend other than re-installing Glance's database. Perhaps the only other solution would to make a tarball of the SQL dump of your glance registry, email me the tarball and I can try to figure out how to uncorrupt it.

-jay

edit flag offensive delete link more
0

answered 2011-07-29 19:18:06 -0500

howard-h gravatar image

Is there no further word on this?

edit flag offensive delete link more
0

answered 2011-07-29 09:16:08 -0500

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

edit flag offensive delete link more
0

answered 2011-07-13 18:32:16 -0500

jaypipes gravatar image

Well, we've made a mess of things, haven't we Howard... ;)

OK, this is a totally separate issue (there's been like 4 separate bugs you've run into in this single question!). I will get with the Glance devs and figure out a solution. This was solved before, but the "fix" for this was removed because it broke other things...

I'll update you as soon as I can, jay

edit flag offensive delete link more
0

answered 2011-07-13 18:02:06 -0500

howard-h gravatar image

Thanks.

I've attempted to install Glance manually now on two different compute nodes but get the error below:

Can you assist, further, please?

sudo add-apt-repository ppa:glance-core/trunk sudo apt-get update sudo apt-get install glance

Reading package lists... Done Building dependency tree
Reading state information... Done The following packages were automatically installed and are no longer required: linux-headers-2.6.38-8-generic linux-headers-2.6.38-8 linux-headers-2.6.38-8-server socat libreadline5 vlan Use 'apt-get autoremove' to remove them. The following extra packages will be installed: python-glance The following packages will be upgraded: glance python-glance 2 upgraded, 0 newly installed, 0 to remove and 34 not upgraded. Need to get 91.5 kB of archives. After this operation, 233 kB of additional disk space will be used. Do you want to continue [Y/n]? Y Get:1 http://ppa.launchpad.net/glance-core/trunk/ubuntu/ (http://ppa.launchpad.net/glance-core/...) natty/main glance all 2011.3~d3~20110709.152-0ubuntu0ppa1~natty1 [14.6 kB] Get:2 http://ppa.launchpad.net/glance-core/trunk/ubuntu/ (http://ppa.launchpad.net/glance-core/...) natty/main python-glance all 2011.3~d3~20110709.152-0ubuntu0ppa1~natty1 [77.0 kB] Fetched 91.5 kB in 0s (442 kB/s)
(Reading database ... 176462 files and directories currently installed.) Preparing to replace glance 2011.2-0ubuntu1 (using .../glance_2011.3~d3~20110709.152-0ubuntu0ppa1~natty1_all.deb) ... glance-registry stop/waiting glance-api stop/waiting Unpacking replacement glance ... Preparing to replace python-glance 2011.2-0ubuntu1 (using .../python-glance_2011.3~d3~20110709.152-0ubuntu0ppa1~natty1_all.deb) ... Unpacking replacement python-glance ... Processing triggers for ureadahead ... Processing triggers for python-support ... Setting up python-glance (2011.3~d3~20110709.152-0ubuntu0ppa1~natty1) ... Setting up glance (2011.3~d3~20110709.152-0ubuntu0ppa1~natty1) ... Installing new version of config file /etc/init/glance-api.conf ... Installing new version of config file /etc/init/glance-registry.conf ... Traceback (most recent call last): File "/usr/bin/glance-manage", line 141, in <module> main() File "/usr/bin/glance-manage", line 137, in main dispatch_cmd(options, args) File "/usr/bin/glance-manage", line 109, in dispatch_cmd cmd_func(options, args) File "/usr/bin/glance-manage", line 97, in do_db_sync glance.registry.db.migration.db_sync(options, version=db_version) File "/usr/lib/pymodules/python2.7/glance/registry/db/migration.py", line 123, in db_sync upgrade(options, version=version) File "/usr/lib/pymodules/python2.7/glance/registry/db/migration.py", line 66, in upgrade return versioning_api.upgrade(sql_connection, repo_path, version) File "/usr/lib/pymodules/python2.7/migrate/versioning/api.py", line 185, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, *opts) File "<string>", line 2, in _migrate File "/usr/lib/pymodules/python2.7/migrate/versioning/util/__init__.py", line 160, in with_engine return f(a, *kw) File "/usr/lib/pymodules/python2.7/migrate/versioning/api.py", line 364, in _migrate schema.runchange(ver, change, changeset.step) File "/usr/lib/pymodules/python2.7/migrate/versioning/schema.py", line 83, in runchange change.run(self.engine, step) File "/usr/lib/pymodules/python2.7/migrate/versioning/script/py.py", line 140, in run script_func(engine) File "/usr/lib/pymodules/python2.7/glance/registry/db/migrate_repo/versions/001_add_images_table.py", line ...

(more)

edit flag offensive delete link more
0

answered 2011-07-13 16:13:07 -0500

jaypipes gravatar image

I don't see any problems with anything in the log. The bind_host is 0.0.0.0 since the glance-api is bound on the localhost of the compute-controller. The log is on that machine...

I have a feeling that what the problem is is that on the compute node (the node where you can't connect properly to Glance from Nova), the old Cactus Glance release is installed (which is the case if you installed Nova but not Glance). Nova's packages have a dependency on the old Glance Cactus package, but the Glance server you are running is on the Diablo package.

I think if you manually install glance on the compute node, things will start working.

I'll file a bug to get the latest Diablo packaging up on PyPI so the Nova installer installs the most up-to-date Glance client package.

edit flag offensive delete link more
0

answered 2011-07-13 07:39:51 -0500

howard-h gravatar image

Jay

The correct curl gives:

curl http://10.0.0.100:9292/v1/images {"images": [{"name": null, "container_format": "ami", "disk_format": "ami", "checksum": "af31c50d70519eba7fb4f52105cb7033", "id": 2, "size": 1476395008}, {"name": null, "container_format": "aki", "disk_format": "aki", "checksum": "d896f8c1577c2a89aa958ab2442817d2", "id": 1, "size": 4106528}]}

I've now got Glance debug logging on and it gives the following after a failed spawn:

2011-07-13 08:32:31 DEBUG [glance-api] ************************* 2011-07-13 08:32:31 DEBUG [glance-api] Configuration options gathered from config file: 2011-07-13 08:32:31 DEBUG [glance-api] /etc/glance/glance-api.conf 2011-07-13 08:32:31 DEBUG [glance-api] ================================================ 2011-07-13 08:32:31 DEBUG [glance-api] bind_host 0.0.0.0 2011-07-13 08:32:31 DEBUG [glance-api] bind_port 9292 2011-07-13 08:32:31 DEBUG [glance-api] debug True 2011-07-13 08:32:31 DEBUG [glance-api] default_store file 2011-07-13 08:32:31 DEBUG [glance-api] filesystem_store_datadir /var/lib/glance/images/ 2011-07-13 08:32:31 DEBUG [glance-api] log_file /var/log/glance/api.log 2011-07-13 08:32:31 DEBUG [glance-api] registry_host 0.0.0.0 2011-07-13 08:32:31 DEBUG [glance-api] registry_port 9191 2011-07-13 08:32:31 DEBUG [glance-api] swift_store_auth_address 127.0.0.1:8080/v1.0/ 2011-07-13 08:32:31 DEBUG [glance-api] swift_store_container glance 2011-07-13 08:32:31 DEBUG [glance-api] swift_store_create_container_on_put False 2011-07-13 08:32:31 DEBUG [glance-api] swift_store_key a86850deb2742ec3cb41518e26aa2d89 2011-07-13 08:32:31 DEBUG [glance-api] swift_store_user jdoe 2011-07-13 08:32:31 DEBUG [glance-api] verbose True 2011-07-13 08:32:31 DEBUG [glance-api] ************************* 2011-07-13 08:32:31 DEBUG [routes.middleware] Initialized with method overriding = True, and path info altering = True 2011-07-13 08:32:31 DEBUG [eventlet.wsgi.server] (18272) wsgi starting up on http://0.0.0.0:9292/ 2011-07-13 08:33:47 DEBUG [glance.api.middleware.version_negotiation] Processing request: HEAD /v1/images/1 Accept: 2011-07-13 08:33:47 DEBUG [glance.api.middleware.version_negotiation] Matched versioned URI. Version: 1.0 2011-07-13 08:33:47 DEBUG [routes.middleware] Matched HEAD /images/1 2011-07-13 08:33:47 DEBUG [routes.middleware] Route path: '/images/{id}', defaults: {'action': u'meta', 'controller': } 2011-07-13 08:33:47 DEBUG [routes.middleware] Match dict: {'action': u'meta', 'controller': , 'id': u'1'} 2011-07-13 08:33:47 DEBUG [eventlet.wsgi.server] 192.168.1.100 - - [13/Jul/2011 08:33:47] "HEAD /v1/images/1 HTTP/1.1" 200 904 0.022955 2011-07-13 08:33:47 DEBUG [glance.api.middleware.version_negotiation] Processing request: HEAD /v1/images/2 Accept: 2011-07-13 08:33:47 DEBUG [glance.api.middleware.version_negotiation] Matched versioned URI. Version: 1.0 2011-07-13 08:33:47 DEBUG [routes.middleware] Matched HEAD /images/2 2011-07-13 08:33:47 DEBUG [routes.middleware] Route path: '/images/{id}', defaults: {'action': u'meta', 'controller': } 2011-07-13 08:33:47 DEBUG [routes.middleware] Match dict: {'action': u'meta', 'controller': , 'id': u'2'} 2011-07-13 08:33:47 DEBUG [eventlet.wsgi.server] 192.168.1.100 - - [13/Jul/2011 08:33:47] "HEAD /v1/images/2 HTTP/1.1" 200 931 0.008692 2011-07-13 08:33:47 DEBUG [glance.api.middleware.version_negotiation] Processing request: HEAD /v1/images/2 Accept: 2011-07-13 08:33:47 DEBUG [glance.api.middleware.version_negotiation] Matched versioned URI. Version: 1.0 ... (more)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-07-09 14:34:21 -0500

Seen: 426 times

Last updated: Aug 11 '11