OCATA cinder-volume error loop
Hi everyone,
We've got a recent error on our opentstack cluster (ocata/ubuntu 1 controller 3 compute nodes) :
The cinder-volume service is configured with 3 backends CEPH (sata sas ssd) The first thread stay down in the cinder service list and generate and error loop in the log related to one missing image/volume. (maybe due to a fail migration with an hard delete from mysql)
2017-04-18 14:12:34.515 15195 INFO cinder.service [-] Starting cinder-volume node (version 10.0.0)
2017-04-18 14:12:34.536 15195 INFO cinder.coordination [-] Coordination backend started successfully.
2017-04-18 14:12:34.537 15195 INFO cinder.volume.manager [req-2c60e3d0-1487-4f34-be56-fd39afc59266 - - - - -] Starting volume driver RBDDriver (1.2.0)
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd [req-2c60e3d0-1487-4f34-be56-fd39afc59266 - - - - -] error opening rbd image volume-b6ce7540-e071-4df8-a700-b37161dcac08
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd Traceback (most recent call last):
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 119, in __init__
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd read_only=read_only)
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd File "rbd.pyx", line 1223, in rbd.Image.__init__ (/tmp/buildd/ceph-11.2.0/obj-x86_64-linux-gnu/src/pybind/rbd/pyrex/rbd.c:11365)
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd ImageNotFound: error opening image volume-b6ce7540-e071-4df8-a700-b37161dcac08 at snapshot None
2017-04-18 14:12:34.603 15195 ERROR cinder.volume.drivers.rbd
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service [req-2c60e3d0-1487-4f34-be56-fd39afc59266 - - - - -] Error starting thread.
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service Traceback (most recent call last):
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service service.start()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 241, in start
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service service_id=Service.service_id)
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 442, in init_host
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service self.driver.init_capabilities()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 719, in init_capabilities
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service stats = self.get_volume_stats(True)
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 423, in get_volume_stats
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service self._update_volume_stats()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 409, in _update_volume_stats
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service self._get_usage_info()
2017-04-18 14:12:34.610 15195 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder ...