Terminate volume connection failed: Parent instance <Volume at 0x7f765fa762d0> is not bound to a Session; lazy load operation of attribute 'volume_admin_metadata' cannot proceed

asked 2016-09-09 23:11:51 -0600

Robin gravatar image

1.The issue happened when we use a script doing “sequential” volume attaching, while they have “many-to-many” (simultaneous attaching many volumes to many VMs) situation. Some info about their infrastructure: operating with 5 physical servers hosing 10 VMs After they setup Hadoop cluster it’s automatically creating 22 volumes simultaneously and simultaneously trying to attach them to 10 VMs. Some VMs are receiving 1 volumes, some 3 and so on. Problem is that from in this case 22 volumes - 3-5 remains not attached. To fix that they have to manually attach them one-by-one.

Similar situation is when they are bringing cluster down – 3-5 volumes remain in “detaching” status. To fix that they have to reset volume status and to detach manually. Volumes, that were impacted by the errors, remain hanging in the detaching state: http://i4.buimg.com/4851/a3fd8b6ce257e6d7.png (image description)

2.log: Logs (cinder-volume):

2016-09-08 10:03:43.355 19276 INFO cinder.volume.manager [req-69ab0a6c-6c2f-4784-8713-ffb5636218e7 1f0a08659d2a4ce1a3523c2637f572be 3b93a86a24e84cd4afae7d336d42fc31 - - -] Terminate volume connection completed successfully. 2016-09-08 10:03:43.357 19276 ERROR cinder.volume.manager [req-ad3defa5-e4ed-48f4-ad1d-70d068390f9e 1f0a08659d2a4ce1a3523c2637f572be 3b93a86a24e84cd4afae7d336d42fc31 - - -] Terminate volume connection failed: Parent instance <volume at="" 0x7f765fa762d0=""> is not bound to a Session; lazy load operation of attribute 'volume_admin_metadata' cannot proceed 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher [req-ad3defa5-e4ed-48f4-ad1d-70d068390f9e 1f0a08659d2a4ce1a3523c2637f572be 3b93a86a24e84cd4afae7d336d42fc31 - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: Parent instance <volume at="" 0x7f765fa762d0=""> is not bound to a Session; lazy load operation of attribute 'volume_admin_metadata' cannot proceed 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, *new_args) 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1541, in terminate_connection 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=err_msg) 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: Parent instance <volume at="" 0x7f765fa762d0=""> is not bound to a Session; lazy load operation of attribute 'volume_admin_metadata' cannot proceed 2016-09-08 10:03:43.358 19276 ERROR oslo_messaging.rpc.dispatcher 2016-09-08 10:03:43.360 19276 ERROR oslo_messaging._drivers.common [req-ad3defa5-e4ed-48f4-ad1d-70d068390f9e 1f0a08659d2a4ce1a3523c2637f572be 3b93a86a24e84cd4afae7d336d42fc31 - - -] Returning exception Bad or unexpected response from the storage volume backend API: Terminate ... (more)

edit retag flag offensive close merge delete