After upgrading Ceph from Firefly to Hammer Cinder Volume is no longer deleting volumes.
Volumes are created fine.
when deleting the following debug output is seen. The cinder volume service then becomes unavailable.
The ceph cluster appears to be operating fine and as before the updates.
Has anyone seen this before? It seems to be cinder-volumes interaction with rbd.
cinder-manage service list
Binary Host Zone Status State Updated At
cinder-scheduler ct nova enabled :-) 2015-04-10 13:35:22
cinder-volume ct nova enabled XXX 2015-04-10 13:24:28
Debug output from cinder-volume
2015-04-10 08:24:38.634 32753 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'69bbba71888641e18ca75bb824e1ba3a', 'tenant': u'8192510bf23e44e9a5dcb7b0558092d1', 'user_identity': u'69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -'}
2015-04-10 08:24:38.637 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got semaphore "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 08:24:38.638 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Attempting to grab file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202
2015-04-10 08:24:38.638 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" at /var/lib/cinder/tmp/cinder-571275f1-550c-4b57-84b6-7598d0f14560-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 08:24:38.661 32753 INFO cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: deleting
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401
2015-04-10 08:24:38.791 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:528
2015-04-10 08:24:38.829 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume volume-571275f1-550c-4b57-84b6-7598d0f14560 is not a clone _get_clone_info /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:551
2015-04-10 08:24:38.914 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] deleting rbd volume volume-571275f1-550c-4b57-84b6-7598d0f14560 delete_volume /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:628