Ask Your Question
0

OCATA cinder-volume error loop

asked 2017-04-18 13:07:18 -0500

lorantino gravatar image

updated 2017-08-20 11:07:37 -0500

sxc731 gravatar image

Hi everyone,

We've got a recent error on our opentstack cluster (ocata/ubuntu 1 controller 3 compute nodes) :

The cinder-volume service is configured with 3 backends CEPH (sata sas ssd) The first thread stay down in the cinder service list and generate and error loop in the log related to one missing image/volume. (maybe due to a fail migration with an hard delete from mysql)

2017-04-18 14:12:34.515 15195 INFO cinder.service [-] Starting cinder-volume node (version 10.0.0)
2017-04-18 14:12:34.536 15195 INFO cinder.coordination [-] Coordination backend started successfully.
2017-04-18 14:12:34.537 15195 INFO cinder.volume.manager [req-2c60e3d0-1487-4f34-be56-fd39afc59266 - - - - -] Starting volume driver RBDDriver (1.2.0)
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd [req-2c60e3d0-1487-4f34-be56-fd39afc59266 - - - - -] error opening rbd image volume-b6ce7540-e071-4df8-a700-b37161dcac08
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd Traceback (most recent call last):
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd   File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 119, in __init__
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd     read_only=read_only)
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd   File "rbd.pyx", line 1223, in rbd.Image.__init__ (/tmp/buildd/ceph-11.2.0/obj-x86_64-linux-gnu/src/pybind/rbd/pyrex/rbd.c:11365)
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd ImageNotFound: error opening image volume-b6ce7540-e071-4df8-a700-b37161dcac08 at snapshot None
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service [req-2c60e3d0-1487-4f34-be56-fd39afc59266 - - - - -] Error starting thread.
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service Traceback (most recent call last):
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service     service.start()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 241, in start
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service     service_id=Service.service_id)
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 442, in init_host
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service     self.driver.init_capabilities()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 719, in init_capabilities
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service     stats = self.get_volume_stats(True)
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 423, in get_volume_stats
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service     self._update_volume_stats()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 409, in _update_volume_stats
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service     self._get_usage_info()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/cinder ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2017-04-19 05:19:45 -0500

lorantino gravatar image

Hi,

Problem have been fixed... we have delete and re create the Ceph pool where Cinder tried to find this volume. Looks like Cinder need to have coherent objects in his data storage, i think an object was corrupted and wasn't a real RBD object.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2017-04-18 12:04:50 -0500

Seen: 684 times

Last updated: Aug 20 '17