Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

ScaleIO volume won't attach to instances

Hi,

I"m running Openstack Queens with ScaleIO 2.6. It's 4 nodes, 1xcontroller and 3xcompute.

I can manually create volumes in ScaleIO and attach them to the compute nodes. So I know the SDCs are configured and working.

I can also create volumes via Openstack. Openstack creates and displays the volume as available and the volume also appears in the ScaleIO Gui.

My problem is, when I try to attach a volume to an instance, it says "attaching" but then returns to "available". Interestingly though, ScaleIO then shows the volume as mapped to the relevant compute node.

The nova-compute.log on the compute has the following in it.

2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Driver failed to attach volume a7c7adec-108e-4635-9555-506fde1f92e1 at /dev/vdb: BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Traceback (most recent call last):
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 539, in _volume_attach
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_type=self['device_type'], encryption=encryption)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1443, in attach_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     encryption=encryption)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in _connect_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     vol_driver.connect_volume(connection_info, instance)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/scaleio.py", line 53, in connect_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_info = self.connector.connect_volume(connection_info['data'])
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 367, in connect_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self.volume_path = self._find_volume_path()
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 91, in _find_volume_path
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     disk_filename = self._wait_for_volume_path(by_id_path)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return r.call(f, *args, **kwargs)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise attempt.get()
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     six.reraise(self.value[0], self.value[1], self.value[2])
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 131, in _wait_for_volume_path
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise exception.BrickException(message=msg)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0] BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Failed to attach a7c7adec-108e-4635-9555-506fde1f92e1 at /dev/vdb: BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Traceback (most recent call last):
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5310, in _attach_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     do_driver_attach=True)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     ret_val = method(obj, context, *args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 619, in attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     virt_driver, do_driver_attach)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 616, in _do_locked_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self._do_attach(*args, **_kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 601, in _do_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     do_driver_attach)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 548, in _volume_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     attachment_id)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self.force_reraise()
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 539, in _volume_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_type=self['device_type'], encryption=encryption)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1443, in attach_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     encryption=encryption)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in _connect_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     vol_driver.connect_volume(connection_info, instance)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/scaleio.py", line 53, in connect_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_info = self.connector.connect_volume(connection_info['data'])
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 367, in connect_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self.volume_path = self._find_volume_path()
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 91, in _find_volume_path
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     disk_filename = self._wait_for_volume_path(by_id_path)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return r.call(f, *args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise attempt.get()
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     six.reraise(self.value[0], self.value[1], self.value[2])
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 131, in _wait_for_volume_path
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise exception.BrickException(message=msg)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0] BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]
2018-07-21 14:00:45.546 2625 ERROR nova.volume.cinder [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] Delete attachment failed for attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e. Error: Volume attachment could not be found with filter: attachment_id = 2dc0279f-8369-4fe3-b39d-9161957b5b1e. (HTTP 404) (Request-ID: req-1be8b72e-fc15-4f05-a606-de5754bbf08c) Code: 404: NotFound: Volume attachment could not be found with filter: attachment_id = 2dc0279f-8369-4fe3-b39d-9161957b5b1e. (HTTP 404) (Request-ID: req-1be8b72e-fc15-4f05-a606-de5754bbf08c)
2018-07-21 14:00:45.546 2625 ERROR root [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5310, in _attach_volume\n    do_driver_attach=True)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped\n    ret_val = method(obj, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 619, in attach\n    virt_driver, do_driver_attach)\n', '  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 616, in _do_locked_attach\n    self._do_attach(*args, **_kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 601, in _do_attach\n    do_driver_attach)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 548, in _volume_attach\n    attachment_id)\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 539, in _volume_attach\n    device_type=self[\'device_type\'], encryption=encryption)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1443, in attach_volume\n    encryption=encryption)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in _connect_volume\n    vol_driver.connect_volume(connection_info, instance)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/scaleio.py", line 53, in connect_volume\n    device_info = self.connector.connect_volume(connection_info[\'data\'])\n', '  File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 367, in connect_volume\n    self.volume_path = self._find_volume_path()\n', '  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 91, in _find_volume_path\n    disk_filename = self._wait_for_volume_path(by_id_path)\n', '  File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper\n    return r.call(f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call\n    raise attempt.get()\n', '  File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n', '  File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n', '  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 131, in _wait_for_volume_path\n    raise exception.BrickException(message=msg)\n', 'BrickException: ScaleIO volume 56cd3f0600000000 not found.\n']: VolumeAttachmentNotFound: Volume attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e could not be found.
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] Exception during message handling: VolumeAttachmentNotFound: Volume attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e could not be found.
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5295, in attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5293, in do_attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     bdm.destroy()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5290, in do_attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5320, in _attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     bdm['attachment_id'])
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 418, in wrapper
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id))
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 450, in _reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     attachment_id)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e could not be found.
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server

Given that volumes are being created and that I can manually map them to the compute nodes, I'm a little unsure where to start and why Openstack won't actually attach them to the instances.

Does anyone have any idea what might be wrong?

ScaleIO volume won't attach to instances

Hi,

I"m running Openstack Queens with ScaleIO 2.6. It's 4 nodes, 1xcontroller and 3xcompute.

I can manually create volumes in ScaleIO and attach them to the compute nodes. So I know the SDCs are configured and working.

I can also create volumes via Openstack. Openstack creates and displays the volume as available and the volume also appears in the ScaleIO Gui.

My problem is, when I try to attach a volume to an instance, it says "attaching" but then returns to "available". Interestingly though, ScaleIO then shows the volume as mapped to the relevant compute node.

The nova-compute.log on the compute has the following in it.

2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Driver failed to attach volume a7c7adec-108e-4635-9555-506fde1f92e1 at /dev/vdb: BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Traceback (most recent call last):
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 539, in _volume_attach
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_type=self['device_type'], encryption=encryption)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1443, in attach_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     encryption=encryption)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in _connect_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     vol_driver.connect_volume(connection_info, instance)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/scaleio.py", line 53, in connect_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_info = self.connector.connect_volume(connection_info['data'])
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 367, in connect_volume
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self.volume_path = self._find_volume_path()
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 91, in _find_volume_path
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     disk_filename = self._wait_for_volume_path(by_id_path)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return r.call(f, *args, **kwargs)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise attempt.get()
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     six.reraise(self.value[0], self.value[1], self.value[2])
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 131, in _wait_for_volume_path
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise exception.BrickException(message=msg)
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0] BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.307 2625 ERROR nova.virt.block_device [instance: 9306343b-4743-4a6d-bf37-5de218914db0]
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Failed to attach a7c7adec-108e-4635-9555-506fde1f92e1 at /dev/vdb: BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0] Traceback (most recent call last):
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5310, in _attach_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     do_driver_attach=True)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     ret_val = method(obj, context, *args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 619, in attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     virt_driver, do_driver_attach)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 616, in _do_locked_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self._do_attach(*args, **_kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 601, in _do_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     do_driver_attach)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 548, in _volume_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     attachment_id)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self.force_reraise()
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 539, in _volume_attach
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_type=self['device_type'], encryption=encryption)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1443, in attach_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     encryption=encryption)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in _connect_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     vol_driver.connect_volume(connection_info, instance)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/scaleio.py", line 53, in connect_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     device_info = self.connector.connect_volume(connection_info['data'])
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return f(*args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 367, in connect_volume
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     self.volume_path = self._find_volume_path()
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 91, in _find_volume_path
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     disk_filename = self._wait_for_volume_path(by_id_path)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     return r.call(f, *args, **kwargs)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise attempt.get()
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     six.reraise(self.value[0], self.value[1], self.value[2])
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 131, in _wait_for_volume_path
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]     raise exception.BrickException(message=msg)
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0] BrickException: ScaleIO volume 56cd3f0600000000 not found.
2018-07-21 14:00:45.522 2625 ERROR nova.compute.manager [instance: 9306343b-4743-4a6d-bf37-5de218914db0]
2018-07-21 14:00:45.546 2625 ERROR nova.volume.cinder [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] Delete attachment failed for attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e. Error: Volume attachment could not be found with filter: attachment_id = 2dc0279f-8369-4fe3-b39d-9161957b5b1e. (HTTP 404) (Request-ID: req-1be8b72e-fc15-4f05-a606-de5754bbf08c) Code: 404: NotFound: Volume attachment could not be found with filter: attachment_id = 2dc0279f-8369-4fe3-b39d-9161957b5b1e. (HTTP 404) (Request-ID: req-1be8b72e-fc15-4f05-a606-de5754bbf08c)
2018-07-21 14:00:45.546 2625 ERROR root [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5310, in _attach_volume\n    do_driver_attach=True)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped\n    ret_val = method(obj, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 619, in attach\n    virt_driver, do_driver_attach)\n', '  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 616, in _do_locked_attach\n    self._do_attach(*args, **_kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 601, in _do_attach\n    do_driver_attach)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 548, in _volume_attach\n    attachment_id)\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 539, in _volume_attach\n    device_type=self[\'device_type\'], encryption=encryption)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1443, in attach_volume\n    encryption=encryption)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in _connect_volume\n    vol_driver.connect_volume(connection_info, instance)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/scaleio.py", line 53, in connect_volume\n    device_info = self.connector.connect_volume(connection_info[\'data\'])\n', '  File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 367, in connect_volume\n    self.volume_path = self._find_volume_path()\n', '  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 91, in _find_volume_path\n    disk_filename = self._wait_for_volume_path(by_id_path)\n', '  File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper\n    return r.call(f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call\n    raise attempt.get()\n', '  File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n', '  File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n', '  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/scaleio.py", line 131, in _wait_for_volume_path\n    raise exception.BrickException(message=msg)\n', 'BrickException: ScaleIO volume 56cd3f0600000000 not found.\n']: VolumeAttachmentNotFound: Volume attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e could not be found.
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server [req-d6a38c5c-921c-43ff-894c-1ffb80a14c5d 8aed461917604d95a5cd7e251660c420 8acb5b594deb4af0b6d54b6823bfe50c - default default] Exception during message handling: VolumeAttachmentNotFound: Volume attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e could not be found.
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5295, in attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5293, in do_attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     bdm.destroy()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5290, in do_attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5320, in _attach_volume
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     bdm['attachment_id'])
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 418, in wrapper
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id))
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 450, in _reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     attachment_id)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment 2dc0279f-8369-4fe3-b39d-9161957b5b1e could not be found.
2018-07-21 14:00:45.599 2625 ERROR oslo_messaging.rpc.server

Given that volumes are being created and that I can manually map them are mapped to the corrent compute nodes, hosts, I'm a little unsure where to start and why Openstack won't actually attach them to the instances.

It does work if I use the lvm volume type. It's only when I use the ScaleIO volume types that it doesn't attach.

Does anyone have any idea what might be wrong?