Ask Your Question
0

Glance image delivery not working on compute nodes

asked 2011-07-09 14:34:21 -0600

howard-h gravatar image

Hi all,

Glance image delivery doesn't seem to be working for me!

I have a three server set-up with Glance running on the cloud controller which also runs compute. Instances that start on the cloud controller work fine but those that attempt to start on the compute nodes always fail. I have the same nova.conf on the nodes and controller. The relevant part of the compute log on the node is:

2011-07-09 15:07:52,811 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/nbd15 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-09 15:07:52,856 DEBUG nova.utils [-] Result was 1 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:166 2011-07-09 15:07:52,857 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -d /dev/nbd15 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-09 15:07:52,906 WARNING nova.virt.libvirt_conn [-] instance instance-00000018: ignoring error injecting data into image 2 (Unexpected error while running command. Command: sudo tune2fs -c 0 -i 0 /dev/nbd15 Exit code: 1 Stdout: 'tune2fs 1.41.14 (22-Dec-2010)\n' Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n") 2011-07-09 15:08:25,952 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last):

Additionally, the files in /var/lib/nova/instances/_base are zero length; I've seen that others have copied the contents of the controller's version of this directory into here but this seems to defeat the object of Glance.

Also, "glance index --host=10.0.0.100" on a node gives the following error (it works fine on the controller):

Failed to show index. Got error: Unknown error occurred! {"versions": [{"status": "CURRENT", "id": "v1.0", "links": [{"href": "http://0.0.0.0:9292/v1/", "rel": "self"}]}]} howard@stratus:~$ ping 192.168.1.44

Can anyone give me some suggestions of a solution or further diagnostic steps--others have posted the same problem but I've not seen a solution.

Thanks

edit retag flag offensive close merge delete

23 answers

Sort by ยป oldest newest most voted
0

answered 2011-08-11 20:33:25 -0600

howard-h gravatar image

I'll give that a go and post an update--I'm using Cactus.

edit flag offensive delete link more
0

answered 2011-07-11 19:53:45 -0600

howard-h gravatar image

glance 2011.3-dev

edit flag offensive delete link more
0

answered 2011-07-11 19:38:50 -0600

jaypipes gravatar image

You can do glance --version.

OK, so the fix for the api.log permissions thing is only in the packaging for Oneric (11.10) it seems. I'll check into it and see if there's a backport to Natty planned.

Cheers, jay

edit flag offensive delete link more
0

answered 2011-07-11 18:36:57 -0600

jaypipes gravatar image

OK, so it looks like you've run into this bug: https://bugs.launchpad.net/glance/+bu... . What version of Glance are you running and on which operating system version?

-jay

edit flag offensive delete link more
0

answered 2011-07-11 19:20:42 -0600

howard-h gravatar image

Ubuntu 11.04; how do I establish the Glance version?

edit flag offensive delete link more
0

answered 2011-07-11 20:25:29 -0600

jaypipes gravatar image

Howard, the fix for the api.log permissions issue is in the bug report above:

touch /var/log/glance/api.log and chown it to the glance user

Are there any files in /var/lib/glance/images? If not, check the permissions of that directory, since when you add images to your Glance installation, by default, that's where Glance will try to write to.

-jay

edit flag offensive delete link more
0

answered 2011-07-11 13:40:13 -0600

blamar gravatar image

Can you please post the entire trace? It seems that the last line of logging you provided was a traceback that could be helpful in debugging this situation. http://paste.openstack.org might be easier for pasting large logs/tracebacks. Thanks!

edit flag offensive delete link more
0

answered 2011-07-11 18:09:15 -0600

howard-h gravatar image

Hi

The complete compute log from on of the failed attempts:

2011-07-11 19:04:50,445 DEBUG nova.rpc [-] received {u'_context_request_id': u'-RIGJJ30OIN-FTNX4TU4', u'_context_read_deleted': False, u'args': {u'instance_id': 26, u'injected_files': None, u'availability_zone': None}, u'_context_is_admin': True, u'_context_timestamp': u'2011-07-11T18:04:49Z', u'_context_user': u'howard', u'method': u'run_instance', u'_context_project': u'cloud5', u'_context_remote_address': u'192.168.1.100'} from (pid=1324) _receive /usr/lib/pymodules/python2.7/nova/rpc.py:167 2011-07-11 19:04:50,447 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-07-11T18:04:49Z', 'remote_address': u'192.168.1.100', 'project': u'cloud5', 'is_admin': True, 'user': u'howard', 'request_id': u'-RIGJJ30OIN-FTNX4TU4', 'read_deleted': False} from (pid=1324) _unpack_context /usr/lib/pymodules/python2.7/nova/rpc.py:331 2011-07-11 19:04:50,535 AUDIT nova.compute.manager [-RIGJJ30OIN-FTNX4TU4 howard cloud5] instance 26: starting... 2011-07-11 19:04:50,750 DEBUG nova.rpc [-] Making asynchronous call on network.nimbus ... from (pid=1324) call /usr/lib/pymodules/python2.7/nova/rpc.py:350 2011-07-11 19:04:50,751 DEBUG nova.rpc [-] MSG_ID is 74c4383a033f4aa5b6255e243481edf9 from (pid=1324) call /usr/lib/pymodules/python2.7/nova/rpc.py:353 2011-07-11 19:04:51,396 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_bridge" for method "ensure_bridge"... from (pid=1324) inner /usr/lib/pymodules/python2.7/nova/utils.py:594 2011-07-11 19:04:51,396 DEBUG nova.utils [-] Attempting to grab file lock "ensure_bridge" for method "ensure_bridge"... from (pid=1324) inner /usr/lib/pymodules/python2.7/nova/utils.py:599 2011-07-11 19:04:51,397 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev br100 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-11 19:04:51,418 DEBUG nova.utils [-] Running cmd (subprocess): sudo route -n from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-11 19:04:51,462 DEBUG nova.utils [-] Running cmd (subprocess): sudo ip addr show dev eth0 scope global from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-11 19:04:51,500 DEBUG nova.utils [-] Running cmd (subprocess): sudo brctl addif br100 eth0 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:150 2011-07-11 19:04:51,534 DEBUG nova.utils [-] Result was 1 from (pid=1324) execute /usr/lib/pymodules/python2.7/nova/utils.py:166 2011-07-11 19:04:51,641 DEBUG nova.virt.libvirt_conn [-] instance instance-0000001a: starting toXML method from (pid=1324) to_xml /usr/lib/pymodules/python2.7/nova/virt/libvirt_conn.py:996 2011-07-11 19:04:51,755 DEBUG nova.virt.libvirt_conn [-] instance instance-0000001a: finished toXML method from (pid=1324) to_xml /usr/lib/pymodules/python2.7/nova/virt/libvirt_conn.py:1041 2011-07-11 19:04:51,878 INFO nova [-] called setup_basic_filtering in nwfilter 2011-07-11 19:04:51,879 INFO nova [-] ensuring static filters 2011-07-11 19:04:51,954 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3997110=""> 2011-07-11 19:04:51,955 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3997cd0=""> 2011-07-11 ... (more)

edit flag offensive delete link more
0

answered 2011-07-11 18:16:45 -0600

jaypipes gravatar image

Hi Howard, could you paste the glance API log please :) Thanks! -jay

edit flag offensive delete link more
0

answered 2011-07-11 18:29:28 -0600

howard-h gravatar image

The Glance API logs on the controller and the node haven't had any additions for a couple of days.

The controller has entries like this--I've changed the permissions on the file but nothing further's been written:

IOError: [Errno 13] Permission denied: '/var/log/glance/api.log' Traceback (most recent call last): File "/usr/bin/glance-api", line 60, in <module> conf, app = config.load_paste_app('glance-api', options, args) File "/usr/lib/pymodules/python2.7/glance/common/config.py", line 279, in load_paste_app setup_logging(options, conf) File "/usr/lib/pymodules/python2.7/glance/common/config.py", line 168, in setup_logging logfile = logging.FileHandler(logfile) File "/usr/lib/python2.7/logging/__init__.py", line 893, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 912, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 13] Permission denied: '/var/log/glance/api.log' Traceback (most recent call last): File "/usr/bin/glance-api", line 60, in <module> conf, app = config.load_paste_app('glance-api', options, args) File "/usr/lib/pymodules/python2.7/glance/common/config.py", line 279, in load_paste_app setup_logging(options, conf) File "/usr/lib/pymodules/python2.7/glance/common/config.py", line 168, in setup_logging logfile = logging.FileHandler(logfile) File "/usr/lib/python2.7/logging/__init__.py", line 893, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 912, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 13] Permission denied: '/var/log/glance/api.log'

the node looks like this at the end:

Session terminated, terminating shell... ...terminated. 2011-07-06 20:30:08 DEBUG [glance-api] ************************* 2011-07-06 20:30:08 DEBUG [glance-api] Configuration options gathered from config file: 2011-07-06 20:30:08 DEBUG [glance-api] /etc/glance/glance.conf 2011-07-06 20:30:08 DEBUG [glance-api] ================================================ 2011-07-06 20:30:08 DEBUG [glance-api] bind_host 0.0.0.0 2011-07-06 20:30:08 DEBUG [glance-api] bind_port 9292 2011-07-06 20:30:08 DEBUG [glance-api] debug False 2011-07-06 20:30:08 DEBUG [glance-api] default_store file 2011-07-06 20:30:08 DEBUG [glance-api] filesystem_store_datadir /var/lib/glance/images/ 2011-07-06 20:30:08 DEBUG [glance-api] registry_host 0.0.0.0 2011-07-06 20:30:08 DEBUG [glance-api] registry_port 9191 2011-07-06 20:30:08 DEBUG [glance-api] swift_store_auth_address 127.0.0.1:8080/v1.0/ 2011-07-06 20:30:08 DEBUG [glance-api] swift_store_container glance 2011-07-06 20:30:08 DEBUG [glance-api] swift_store_create_container_on_put False 2011-07-06 20:30:08 DEBUG [glance-api] swift_store_key a86850deb2742ec3cb41518e26aa2d89 2011-07-06 20:30:08 DEBUG [glance-api] swift_store_user jdoe 2011-07-06 20:30:08 DEBUG [glance-api] verbose True 2011-07-06 20:30:08 DEBUG [glance-api] ************************* 2011-07-06 20:30:08 DEBUG [routes.middleware] Initialized with method overriding = True, and path info altering = True 2011-07-06 20:30:08 DEBUG [eventlet.wsgi.server] (1320) wsgi starting up on http://0.0.0.0:9292/ 2011-07-09 12:10:15 DEBUG [routes.middleware] Matched GET /images/1 2011-07-09 12:10:15 DEBUG [routes.middleware] Route path: '/images/:(id)', defaults: {'action': u'show', 'controller': } 2011-07-09 12:10:15 DEBUG [routes.middleware] Match dict: {'action': u'show', 'controller': , 'id': u'1 ... (more)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-07-09 14:34:21 -0600

Seen: 434 times

Last updated: Aug 11 '11