Revision history [back]

After upgrading Ceph from Firefly to Hammer Cinder Volume is no longer deleting volumes.
Volumes are created fine.

when deleting the following debug output is seen. The cinder volume service then becomes unavailable.

The ceph cluster appears to be operating fine and as before the updates.

Has anyone seen this before? It seems to be cinder-volumes interaction with rbd.

cinder-manage service list

Binary           Host                                 Zone             Status     State Updated At
cinder-scheduler ct                                   nova             enabled    :-)   2015-04-10 13:35:22
cinder-volume    ct                                   nova             enabled    XXX   2015-04-10 13:24:28

Debug output from cinder-volume

    2015-04-10 08:24:38.634 32753 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'69bbba71888641e18ca75bb824e1ba3a', 'tenant': u'8192510bf23e44e9a5dcb7b0558092d1', 'user_identity': u'69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -'}
2015-04-10 08:24:38.637 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got semaphore "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 08:24:38.638 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Attempting to grab file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202
2015-04-10 08:24:38.638 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" at /var/lib/cinder/tmp/cinder-571275f1-550c-4b57-84b6-7598d0f14560-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 08:24:38.661 32753 INFO cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: deleting
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401
2015-04-10 08:24:38.791 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:528
2015-04-10 08:24:38.829 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume volume-571275f1-550c-4b57-84b6-7598d0f14560 is not a clone _get_clone_info /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:551
2015-04-10 08:24:38.914 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] deleting rbd volume volume-571275f1-550c-4b57-84b6-7598d0f14560 delete_volume /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:628

After upgrading Ceph from Firefly to Hammer Cinder Volume is no longer deleting volumes.
Volumes are created fine.

when deleting the following debug output is seen. The cinder volume service then becomes unavailable.

The ceph cluster appears to be operating fine and as before the updates.

Has anyone seen this before? It seems to be cinder-volumes interaction with rbd.

cinder-manage service list

Binary           Host                     Zone             Status     State Updated At
cinder-scheduler ct                   nova             enabled    :-)   2015-04-10 13:35:22
cinder-volume    ct                    nova             enabled    XXX   2015-04-10 13:24:28

Debug output from cinder-volume

    2015-04-10 08:24:38.634 32753 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'69bbba71888641e18ca75bb824e1ba3a', 'tenant': u'8192510bf23e44e9a5dcb7b0558092d1', 'user_identity': u'69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -'}
2015-04-10 08:24:38.637 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got semaphore "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 08:24:38.638 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Attempting to grab file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202
2015-04-10 08:24:38.638 32753 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" at /var/lib/cinder/tmp/cinder-571275f1-550c-4b57-84b6-7598d0f14560-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 08:24:38.661 32753 INFO cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: deleting
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401
2015-04-10 08:24:38.791 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:528
2015-04-10 08:24:38.829 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume volume-571275f1-550c-4b57-84b6-7598d0f14560 is not a clone _get_clone_info /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:551
2015-04-10 08:24:38.914 32753 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] deleting rbd volume volume-571275f1-550c-4b57-84b6-7598d0f14560 delete_volume /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:628

After upgrading Ceph from Firefly to Hammer Cinder Volume is no longer deleting volumes.
Volumes are created fine.

when deleting the following debug output is seen. The cinder volume service then becomes unavailable.

The ceph cluster appears to be operating fine and as before the updates.

Has anyone seen this before? It seems to be cinder-volumes interaction with rbd.

cinder-manage service list

Binary           Host                    Zone             Status     State Updated At
cinder-scheduler ct                  nova             enabled    :-)   2015-04-10 13:35:22
cinder-volume    ct                   nova             enabled    XXX   2015-04-10 13:24:28

Debug output from cinder-volume

    2015-04-10 08:24:38.634 32753 2015-04-10 08:56:44.482 5527 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'69bbba71888641e18ca75bb824e1ba3a', 'tenant': u'8192510bf23e44e9a5dcb7b0558092d1', 'user_identity': u'69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -'}
2015-04-10 08:24:38.637 32753 08:56:44.482 5527 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got semaphore "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 08:24:38.638 32753 08:56:44.483 5527 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Attempting to grab file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202
2015-04-10 08:24:38.638 32753 08:56:44.483 5527 DEBUG cinder.openstack.common.lockutils [req-e5fb8e10-d41b-4d92-9290-792d93ff331a [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got file lock "571275f1-550c-4b57-84b6-7598d0f14560-delete_volume" "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" at /var/lib/cinder/tmp/cinder-571275f1-550c-4b57-84b6-7598d0f14560-delete_volume /var/lib/cinder/tmp/cinder-8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 08:24:38.661 32753 08:56:44.506 5527 INFO cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: 08:56:44.507 5527 DEBUG cinder.volume.manager [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 08:24:38.662 32753 DEBUG cinder.volume.manager [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 571275f1-550c-4b57-84b6-7598d0f14560: 08:56:44.507 5527 DEBUG cinder.volume.manager [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401
2015-04-10 08:24:38.791 32753 

2015-04-10 08:57:05.397 5527 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:528
2015-04-10 08:24:38.829 32753 08:57:05.413 5527 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume volume-571275f1-550c-4b57-84b6-7598d0f14560 [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume volume-8ad3c07b-5947-4376-b2b8-e7c608e35048 is not a clone _get_clone_info /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:551
2015-04-10 08:24:38.914 32753 08:57:05.446 5527 DEBUG cinder.volume.drivers.rbd [req-e5fb8e10-d41b-4d92-9290-792d93ff331a [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] deleting rbd volume volume-571275f1-550c-4b57-84b6-7598d0f14560 volume-8ad3c07b-5947-4376-b2b8-e7c608e35048 delete_volume /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:628

2015-04-10 09:08:17.503 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Re-exporting 3 volumes init_host /usr/lib/python2.6/site-packages/cinder/volume/manager.py:254
2015-04-10 09:08:17.503 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 5862adc9-3ed3-4afc-8091-2d981dfa55eb: skipping export
2015-04-10 09:08:17.504 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: skipping export
2015-04-10 09:08:17.504 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Resuming any in progress delete operations init_host /usr/lib/python2.6/site-packages/cinder/volume/manager.py:293
2015-04-10 09:08:17.504 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Resuming delete on volume: 8ad3c07b-5947-4376-b2b8-e7c608e35048
2015-04-10 09:08:17.505 11651 DEBUG cinder.openstack.common.lockutils [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Got semaphore "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 09:08:17.505 11651 DEBUG cinder.openstack.common.lockutils [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Attempting to grab file lock "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202


2015-04-10 09:09:31.462 5527 INFO cinder.openstack.common.service [-] Parent process has died unexpectedly, exiting
2015-04-10 09:09:31.467 5527 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.6/site-packages/cinder/openstack/common/periodic_task.py:178
2015-04-10 09:09:31.467 5527 DEBUG cinder.manager [-] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.6/site-packages/cinder/manager.py:128
2015-04-10 09:09:31.535 11651 DEBUG cinder.openstack.common.lockutils [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Got file lock "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" at /var/lib/cinder/tmp/cinder-8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 09:09:31.556 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting
2015-04-10 09:09:31.557 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 09:09:31.557 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401
2015-04-10 09:09:31.615 11651 INFO cinder.volume.drivers.rbd [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume volume-8ad3c07b-5947-4376-b2b8-e7c608e35048 no longer exists in backend

After upgrading Ceph from Firefly to Hammer Cinder Volume is no longer deleting volumes.
Volumes are created fine.

when deleting the following debug output is seen. The cinder volume service then becomes unavailable.

The ceph cluster appears to be operating fine and as before the updates.

Has anyone seen this before? It seems to be cinder-volumes interaction with rbd.

cinder-manage service list

Binary           Host                    Zone             Status     State Updated At
cinder-scheduler ct                  nova             enabled    :-)   2015-04-10 13:35:22
cinder-volume    ct                   nova             enabled    XXX   2015-04-10 13:24:28

Debug output from cinder-volume

2015-04-10 08:56:44.482 5527 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'69bbba71888641e18ca75bb824e1ba3a', 'tenant': u'8192510bf23e44e9a5dcb7b0558092d1', 'user_identity': u'69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -'}
2015-04-10 08:56:44.482 5527 DEBUG cinder.openstack.common.lockutils [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got semaphore "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 08:56:44.483 5527 DEBUG cinder.openstack.common.lockutils [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Attempting to grab file lock "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202
2015-04-10 08:56:44.483 5527 DEBUG cinder.openstack.common.lockutils [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] Got file lock "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" at /var/lib/cinder/tmp/cinder-8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 08:56:44.506 5527 INFO cinder.volume.manager [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting
2015-04-10 08:56:44.507 5527 DEBUG cinder.volume.manager [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 08:56:44.507 5527 DEBUG cinder.volume.manager [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401


2015-04-10 08:57:05.397 5527 DEBUG cinder.volume.drivers.rbd [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:528
2015-04-10 08:57:05.413 5527 DEBUG cinder.volume.drivers.rbd [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] volume volume-8ad3c07b-5947-4376-b2b8-e7c608e35048 is not a clone _get_clone_info /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:551
2015-04-10 08:57:05.446 5527 DEBUG cinder.volume.drivers.rbd [req-88f65138-74d5-48e0-a06a-8fd8bbc1cdeb 69bbba71888641e18ca75bb824e1ba3a 8192510bf23e44e9a5dcb7b0558092d1 - - -] deleting rbd volume volume-8ad3c07b-5947-4376-b2b8-e7c608e35048 delete_volume /usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py:628

After a restart of cinder-volume

2015-04-10 09:08:17.503 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Re-exporting 3 volumes init_host /usr/lib/python2.6/site-packages/cinder/volume/manager.py:254
2015-04-10 09:08:17.503 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 5862adc9-3ed3-4afc-8091-2d981dfa55eb: skipping export
2015-04-10 09:08:17.504 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: skipping export
2015-04-10 09:08:17.504 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Resuming any in progress delete operations init_host /usr/lib/python2.6/site-packages/cinder/volume/manager.py:293
2015-04-10 09:08:17.504 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Resuming delete on volume: 8ad3c07b-5947-4376-b2b8-e7c608e35048
2015-04-10 09:08:17.505 11651 DEBUG cinder.openstack.common.lockutils [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Got semaphore "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2015-04-10 09:08:17.505 11651 DEBUG cinder.openstack.common.lockutils [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Attempting to grab file lock "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202


2015-04-10 09:09:31.462 5527 INFO cinder.openstack.common.service [-] Parent process has died unexpectedly, exiting
2015-04-10 09:09:31.467 5527 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.6/site-packages/cinder/openstack/common/periodic_task.py:178
2015-04-10 09:09:31.467 5527 DEBUG cinder.manager [-] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.6/site-packages/cinder/manager.py:128
2015-04-10 09:09:31.535 11651 DEBUG cinder.openstack.common.lockutils [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] Got file lock "8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume" at /var/lib/cinder/tmp/cinder-8ad3c07b-5947-4376-b2b8-e7c608e35048-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2015-04-10 09:09:31.556 11651 INFO cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting
2015-04-10 09:09:31.557 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: removing export delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:399
2015-04-10 09:09:31.557 11651 DEBUG cinder.volume.manager [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume 8ad3c07b-5947-4376-b2b8-e7c608e35048: deleting delete_volume /usr/lib/python2.6/site-packages/cinder/volume/manager.py:401
2015-04-10 09:09:31.615 11651 INFO cinder.volume.drivers.rbd [req-15123499-df0d-4fce-b66b-01dd6fea665b - - - - -] volume volume-8ad3c07b-5947-4376-b2b8-e7c608e35048 no longer exists in backend