Revision history [back]

click to hide/show revision 1
initial version

Migration of attached volumes fails

My openstack cluster uses cinder with the rbd driver for ceph backed storage for my volumes. I have two separate rbd pools, one for HDD storage and another for SSD storage. I am able to migrate volumes between pools, unless the are attached. I can provide logs and configs if needed.

Migration of attached volumes fails

My openstack cluster uses cinder with the rbd driver for ceph backed storage for my volumes. I have two separate rbd pools, storage pools on shared storage. When I try to migrate a volume with cinder from one for HDD storage pool to another, everything works as expected; unless the volume is attached. If I try to migrate an attached volume: It creates the new volume on the 2nd pool, spits out a large error in the logs, and another for SSD storage. then deletes the volume from the 2nd pool. This is the error I am able seeing:

2017-12-13 09:44:29.526 2171 INFO cinder.volume.flows.manager.create_volume [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Volume ed416399-f356-4d65-9b6b-c4bf2c4efe6d: being created as raw with specification: {'status': u'creating', 'volume_size': 10, 'volume_name': 'volume-ed416399-f356-4d65-9b6b-c4bf2c4efe6d'}
2017-12-13 09:44:29.917 2171 INFO cinder.volume.flows.manager.create_volume [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Volume volume-ed416399-f356-4d65-9b6b-c4bf2c4efe6d (ed416399-f356-4d65-9b6b-c4bf2c4efe6d): created successfully
2017-12-13 09:44:29.925 2171 INFO cinder.volume.manager [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Created volume successfully.
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Failed to migrate volumes between pools, unless the are attached. I can provide logs and configs if needed.

copy volume dc06f100-af7f-4124-b4f6-8f1a7e6a95a4 to ed416399-f356-4d65-9b6b-c4bf2c4efe6d 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager Traceback (most recent call last): 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1788, in _migrate_volume_generic 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager new_volume.id) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 181, in update_server_volume 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager new_volume_id) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/novaclient/v2/volumes.py", line 68, in update_server_volume 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager body, "volumeAttachment") 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 370, in _update 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager resp, body = self.api.client.put(url, body=body) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 196, in put 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return self.request(url, 'PUT', **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 107, in request 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 344, in request 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 112, in request 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return self.session.request(url, method, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return wrapped(*args, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 462, in request 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager auth_headers = self.get_auth_headers(auth) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 727, in get_auth_headers 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return auth.get_headers(self, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager token = self.get_token(session) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return self.get_access(session).auth_token 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager self.auth_ref = self.get_auth_ref(session) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 179, in get_auth_ref 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager self._plugin = self._do_create_plugin(session) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 118, in _do_create_plugin 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager authenticated=False) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return wrapped(*args, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 319, in get_discovery 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager authenticated=authenticated) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return wrapped(*args, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 140, in __init__ 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager authenticated=authenticated) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return wrapped(*args, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 40, in get_version_data 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager resp = session.get(url, headers=headers, authenticated=authenticated) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 667, in get 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return self.request(url, 'GET', **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager return wrapped(*args, **kwargs) 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 487, in request 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager raise exceptions.EndpointNotFound() 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager EndpointNotFound: Could not find requested endpoint in Service Catalog. 2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Exception during message handling 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 4427, in migrate_volume 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server new_type_id=new_type_id, volume=volume) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1990, in migrate_volume 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server volume.save() 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1983, in migrate_volume 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server new_type_id) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1795, in _migrate_volume_generic 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server new_volume) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1788, in _migrate_volume_generic 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server new_volume.id) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 181, in update_server_volume 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server new_volume_id) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/novaclient/v2/volumes.py", line 68, in update_server_volume 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server body, "volumeAttachment") 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 370, in _update 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server resp, body = self.api.client.put(url, body=body) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 196, in put 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 107, in request 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 344, in request 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 112, in request 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return wrapped(*args, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 462, in request 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 727, in get_auth_headers 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server token = self.get_token(session) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 179, in get_auth_ref 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server self._plugin = self._do_create_plugin(session) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 118, in _do_create_plugin 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server authenticated=False) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return wrapped(*args, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 319, in get_discovery 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server authenticated=authenticated) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return wrapped(*args, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 140, in __init__ 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server authenticated=authenticated) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return wrapped(*args, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 40, in get_version_data 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server resp = session.get(url, headers=headers, authenticated=authenticated) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 667, in get 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server return wrapped(*args, **kwargs) 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 487, in request 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server raise exceptions.EndpointNotFound() 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server EndpointNotFound: Could not find requested endpoint in Service Catalog. 2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server 2017-12-13 09:44:31.129 2171 INFO cinder.volume.manager [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Deleted volume successfully.

Migration of attached volumes fails

I have two storage pools on shared storage. When I try to migrate a volume with cinder from one pool to another, everything works as expected; unless the volume is attached. If I try to migrate an attached volume: It creates the new volume on the 2nd pool, spits out a large error in the logs, and then deletes the volume from the 2nd pool. This is the error I am seeing:

2017-12-13 09:44:29.526 2171 INFO cinder.volume.flows.manager.create_volume [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Volume ed416399-f356-4d65-9b6b-c4bf2c4efe6d: being created as raw with specification: {'status': u'creating', 'volume_size': 10, 'volume_name': 'volume-ed416399-f356-4d65-9b6b-c4bf2c4efe6d'}
2017-12-13 09:44:29.917 2171 INFO cinder.volume.flows.manager.create_volume [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Volume volume-ed416399-f356-4d65-9b6b-c4bf2c4efe6d (ed416399-f356-4d65-9b6b-c4bf2c4efe6d): created successfully
2017-12-13 09:44:29.925 2171 INFO cinder.volume.manager [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Created volume successfully.
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Failed to copy volume dc06f100-af7f-4124-b4f6-8f1a7e6a95a4 to ed416399-f356-4d65-9b6b-c4bf2c4efe6d
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager Traceback (most recent call last):
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1788, in _migrate_volume_generic
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     new_volume.id)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 181, in update_server_volume
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     new_volume_id)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/novaclient/v2/volumes.py", line 68, in update_server_volume
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     body, "volumeAttachment")
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 370, in _update
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     resp, body = self.api.client.put(url, body=body)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 196, in put
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return self.request(url, 'PUT', **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 107, in request
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 344, in request
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 112, in request
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return self.session.request(url, method, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 462, in request
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     auth_headers = self.get_auth_headers(auth)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 727, in get_auth_headers
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return auth.get_headers(self, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     token = self.get_token(session)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return self.get_access(session).auth_token
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     self.auth_ref = self.get_auth_ref(session)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 179, in get_auth_ref
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     self._plugin = self._do_create_plugin(session)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 118, in _do_create_plugin
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     authenticated=False)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 319, in get_discovery
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     authenticated=authenticated)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 140, in __init__
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     authenticated=authenticated)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 40, in get_version_data
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     resp = session.get(url, headers=headers, authenticated=authenticated)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 667, in get
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return self.request(url, 'GET', **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 487, in request
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager     raise exceptions.EndpointNotFound()
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager EndpointNotFound: Could not find requested endpoint in Service Catalog.
2017-12-13 09:44:30.678 2256 ERROR cinder.volume.manager 
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Exception during message handling
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 4427, in migrate_volume
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     new_type_id=new_type_id, volume=volume)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1990, in migrate_volume
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     volume.save()
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1983, in migrate_volume
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     new_type_id)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1795, in _migrate_volume_generic
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     new_volume)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1788, in _migrate_volume_generic
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     new_volume.id)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 181, in update_server_volume
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     new_volume_id)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/novaclient/v2/volumes.py", line 68, in update_server_volume
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     body, "volumeAttachment")
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 370, in _update
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.put(url, body=body)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 196, in put
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return self.request(url, 'PUT', **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 107, in request
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 344, in request
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 112, in request
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return self.session.request(url, method, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 462, in request
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     auth_headers = self.get_auth_headers(auth)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 727, in get_auth_headers
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return auth.get_headers(self, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     token = self.get_token(session)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return self.get_access(session).auth_token
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     self.auth_ref = self.get_auth_ref(session)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 179, in get_auth_ref
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     self._plugin = self._do_create_plugin(session)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 118, in _do_create_plugin
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     authenticated=False)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 319, in get_discovery
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     authenticated=authenticated)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 140, in __init__
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     authenticated=authenticated)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 40, in get_version_data
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     resp = session.get(url, headers=headers, authenticated=authenticated)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 667, in get
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return self.request(url, 'GET', **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     return wrapped(*args, **kwargs)
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 487, in request
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server     raise exceptions.EndpointNotFound()
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server EndpointNotFound: Could not find requested endpoint in Service Catalog.
2017-12-13 09:44:30.857 2256 ERROR oslo_messaging.rpc.server 
2017-12-13 09:44:31.129 2171 INFO cinder.volume.manager [req-e53bd700-d327-4787-a7af-cce4b8f955f4 0f2400d4b85341dea4481382e49eab41 0bc21ce6de3f4870b0f4f68148956053 - default default] Deleted volume successfully.