Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Failed to attach volume to instance

I've just installed OpenStack Ocata (via RDO), then I created a volume, an instance, and attached the volume to the instance.

After a sudden power surge, the controller got rebooted. Now, the volume is not attached to the instance anymore, and I can't attach it using the following command (or the web interface):

nova volume-attach 5f3033e1-55a3-4962-bd5e-d242462a6380 2567c36c-7897-43ea-b27a-bad2d00ec105 auto

I tried to create a new volume and managed to attach it succesfully. For testing purposes, I forced the controller to shutdown, but the same problem happened again.

Cinder's volume.log shows the following error:

2017-05-15 09:47:53.604 22417 INFO cinder.volume.targets.lio [req-329b83a5-9396-4ed6-aba2-d8fce8bf3252 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Creating iscsi_target for volume: volume-2567c36c-7897-43ea-b27a-bad2d00ec105
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio [req-329b83a5-9396-4ed6-aba2-d8fce8bf3252 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Failed to create iscsi target for volume id:volume-2567c36c-7897-43ea-b27a-bad2d00ec105.
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio Traceback (most recent call last):
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 127, in create_iscsi_target
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio     self._execute(*command_args, run_as_root=True)
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio     return f(*args, **kwargs)
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 52, in _execute
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio     return utils.execute(*args, **kwargs)
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio     return processutils.execute(*cmd, **kwargs)
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio     cmd=sanitized_cmd)
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio ProcessExecutionError: Unexpected error while running command.
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool create /dev/cinder-volumes/volume-2567c36c-7897-43ea-b27a-bad2d00ec105 iqn.2010-10.org.openstack:volume-2567c36c-7897-43ea-b27a-bad2d00ec105 nBU4v8TsSCU5H6di7dYC f9TdkwtKXbUWgSR8 False -p3260 -a192.168.0.208
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio Exit code: 1
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio Stdout: u''
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio Stderr: u'Traceback (most recent call last):\n  File "/bin/cinder-rtstool", line 10, in <module>\n    sys.exit(main())\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 293, in main\n    **optional_args)\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 61, in create\n    dev=backing_device)\n  File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 672, in __init__\n    self._configure(dev, wwn, readonly)\n  File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 682, in _configure\n    raise RTSLibError("Device %s is not a TYPE_DISK block device" % dev)\nrtslib_fb.utils.RTSLibError: Device /dev/cinder-volumes/volume-2567c36c-7897-43ea-b27a-bad2d00ec105 is not a TYPE_DISK block device\n'
2017-05-15 09:47:54.189 22417 ERROR cinder.volume.targets.lio 
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager [req-329b83a5-9396-4ed6-aba2-d8fce8bf3252 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Create export of volume failed (Failed to create iscsi target for volume volume-2567c36c-7897-43ea-b27a-bad2d00ec105.)
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager Traceback (most recent call last):
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1494, in initialize_connection
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager     volume, connector)
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 833, in create_export
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager     volume_path)
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager     **portals_config)
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 132, in create_iscsi_target
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager     raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-2567c36c-7897-43ea-b27a-bad2d00ec105.
2017-05-15 09:47:54.190 22417 ERROR cinder.volume.manager 
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server [req-329b83a5-9396-4ed6-aba2-d8fce8bf3252 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Exception during message handling
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1498, in initialize_connection
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server     raise exception.VolumeBackendAPIException(data=msg)
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export of volume failed (Failed to create iscsi target for volume volume-2567c36c-7897-43ea-b27a-bad2d00ec105.)
2017-05-15 09:47:54.191 22417 ERROR oslo_messaging.rpc.server 
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio [req-49e3b2e8-dba5-498b-91e4-45a6c8c170a0 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Failed to delete initiator iqn iqn.1994-05.com.redhat:fc204ed4dc47 from target.
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio Traceback (most recent call last):
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 197, in terminate_connection
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio     run_as_root=True)
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio     return f(*args, **kwargs)
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 52, in _execute
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio     return utils.execute(*args, **kwargs)
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio     return processutils.execute(*cmd, **kwargs)
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio     cmd=sanitized_cmd)
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio ProcessExecutionError: Unexpected error while running command.
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool delete-initiator iqn.2010-10.org.openstack:volume-2567c36c-7897-43ea-b27a-bad2d00ec105 iqn.1994-05.com.redhat:fc204ed4dc47
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio Exit code: 1
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio Stdout: u''
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio Stderr: u'Traceback (most recent call last):\n  File "/bin/cinder-rtstool", line 10, in <module>\n    sys.exit(main())\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 313, in main\n    delete_initiator(target_iqn, initiator_iqn)\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 143, in delete_initiator\n    target = _lookup_target(target_iqn, initiator_iqn)\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 123, in _lookup_target\n    raise RtstoolError(_(\'Could not find target %s\') % target_iqn)\ncinder.cmd.rtstool.RtstoolError: Could not find target iqn.2010-10.org.openstack:volume-2567c36c-7897-43ea-b27a-bad2d00ec105\n'
2017-05-15 09:47:54.896 22417 ERROR cinder.volume.targets.lio 
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager [req-49e3b2e8-dba5-498b-91e4-45a6c8c170a0 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Terminate volume connection failed: Failed to detach iSCSI target for volume 2567c36c-7897-43ea-b27a-bad2d00ec105.
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager Traceback (most recent call last):
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1537, in terminate_connection
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager     force=force)
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 848, in terminate_connection
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager     **kwargs)
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 202, in terminate_connection
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager     raise exception.ISCSITargetDetachFailed(volume_id=volume['id'])
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager ISCSITargetDetachFailed: Failed to detach iSCSI target for volume 2567c36c-7897-43ea-b27a-bad2d00ec105.
2017-05-15 09:47:54.897 22417 ERROR cinder.volume.manager 
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server [req-49e3b2e8-dba5-498b-91e4-45a6c8c170a0 acfeb5bc5d6b470a9a0c2c9541a45778 e5a510637914482eadef78c8a6bd5e3a - default default] Exception during message handling
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1542, in terminate_connection
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server     raise exception.VolumeBackendAPIException(data=err_msg)
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: Failed to detach iSCSI target for volume 2567c36c-7897-43ea-b27a-bad2d00ec105.
2017-05-15 09:47:54.898 22417 ERROR oslo_messaging.rpc.server