Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Cannot attach volume - cinder + netapp + iscsi

I am able to create a volume on the NetApp SAN from horizon and the command line, but I am not able to attach that volume to any instances. I can attach a LVM volume to instances.

I'm on newton. NetApp dataontap 7-mode iscsi

The luns are created correctly so cinder can get into the netapp and configure things. I've checked cinder.conf for the ordinary things listed in other posts.

openstack server list | grep x_db
| 80bde0e8-1c17-4c77-add1-49cfb75e19f7 | x_db    | ACTIVE | admin_internal_net=192.168.111.9                 |

openstack volume list
+--------------------------------------+--------------+-----------+------+---------------------------------------+
| ID                                   | Display Name | Status    | Size | Attached to                           |
+--------------------------------------+--------------+-----------+------+---------------------------------------+
| 723dd3bc-d67a-47e7-8cb7-32ddce80f810 | med_test     | available |    3 |                                       |
| dd76afad-ff33-4379-ab48-342c0c8674b0 | test_vol     | in-use    |    2 | Attached to x_db on /dev/vdb  |
+--------------------------------------+--------------+-----------+------+---------------------------------------+

And the command to attach the volume:

#openstack --debug server add volume 80bde0e8-1c17-4c77-add1-49cfb75e19f7 723dd3bc-d67a-47e7-8cb7-32ddce80f810

https://gist.github.com/solvire/ec8ad7910b138c1f8cd7c26ec9bd9ce0

The nova logs from the fuel server:

nova.compute.manager [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Attaching volume 723dd3bc-d67a-47e7-8cb7-32ddce80f810 to /dev/vdc

os_brick.initiator.connectors.iscsi [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] Multipath discovery for iSCSI enabled

nova.virt.block_device [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Driver failed to attach volume 723dd3bc-d67a-47e7-8cb7-32ddce80f810 at /dev/vdc 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Traceback (most recent call last): 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] device_type=self['device_type'], encryption=encryption) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] self._connect_volume(connection_info, disk_info) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] vol_driver.connect_volume(connection_info, disk_info) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] device_info = self.connector.connect_volume(connection_info['data']) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device 

nova.compute.manager [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Failed to attach 723dd3bc-d67a-47e7-8cb7-32ddce80f810 at /dev/vdc 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Traceback (most recent call last): 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4679, in _attach_volume 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] do_check_attach=False, do_driver_attach=True) 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] ret_val = method(obj, context, *args, **kwargs) 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] connector) 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] self.force_reraise() 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-21 02:00:40.564 130

oslo_messaging.rpc.server [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] Exception during message handling 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py",

cinder.conf

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = pkgosvg0
iscsi_protocol = iscsi
iscsi_helper = tgtadm

[LVM-backend]
volume_backend_name=LVM-backend
volumes_dir=/var/lib/cinder/volumes
iscsi_helper=tgtadm
volume_group=cinder
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
iscsi_ip_address=192.168.1.3

[cinder_netapp]
netapp_login=auser
netapp_controller_ips=
netapp_lun_ostype=linux
netapp_vserver=
netapp_server_port=
nfs_shares_config=/etc/cinder/shares.conf
netapp_sa_password=
thres_avl_size_perc_start=20
reserved_percentage=0
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_enable_multiattach=False
netapp_storage_protocol=iscsi
max_oversubscription_ratio=1.0
backend_host=str:netapp
netapp_storage_family=ontap_7mode
volume_backend_name=cinder_netapp
nfs_mount_options=
thres_avl_size_perc_stop=60
netapp_vfiler=
netapp_server_hostname=192.168.1.xx2
netapp_host_type=linux
expiry_thres_minutes=720
netapp_pool_name_search_pattern=(vol_openstack_junk.+)
netapp_transport_type=http
netapp_password=apswd
netapp_webservice_path=/devmgr/v2
netapp_lun_space_reservation=enabled
use_multipath_for_image_xfer=True
netapp_copyoffload_tool_path=

tgt daemon is running on both controller and compute node

service tgt status
● tgt.service - (i)SCSI target daemon
   Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-10-20 23:06:39 UTC; 3h 41min ago

From the netapp:

I removed all the initiators from the access list.

netapp01> iscsi interface accesslist show
No interface access lists

I can see the netapp being managed properly. (seemingly)

Fri Oct 21 01:37:02 GMT [netapp01:lun.map:info]: LUN /vol/vol_openstack_junk/volume-723dd3bc-d67a-47e7-8cb7-32ddce80f810 was mapped to initiator group openstack-509e4a19-8fb8-4e35-b276-a49bd54d5e7f=0  
Fri Oct 21 01:38:37 GMT [netapp01:lun.map.unmap:info]: LUN /vol/vol_openstack_junk/volume-723dd3bc-d67a-47e7-8cb7-32ddce80f810 unmapped from initiator group openstack-509e4a19-8fb8-4e35-b276-a49bd54d5e7f

I'm about 2 days into this one. Any help is greatly appreciated.

Cannot attach volume - cinder + netapp + iscsi

I am able to create a volume on the NetApp SAN from horizon and the command line, but I am not able to attach that volume to any instances. I can attach a LVM volume to instances.

I'm on newton. NetApp dataontap 7-mode iscsi

The luns are created correctly so cinder can get into the netapp and configure things. I've checked cinder.conf for the ordinary things listed in other posts.

openstack server list | grep x_db
| 80bde0e8-1c17-4c77-add1-49cfb75e19f7 | x_db    | ACTIVE | admin_internal_net=192.168.111.9                 |

openstack volume list
+--------------------------------------+--------------+-----------+------+---------------------------------------+
| ID                                   | Display Name | Status    | Size | Attached to                           |
+--------------------------------------+--------------+-----------+------+---------------------------------------+
| 723dd3bc-d67a-47e7-8cb7-32ddce80f810 | med_test     | available |    3 |                                       |
| dd76afad-ff33-4379-ab48-342c0c8674b0 | test_vol     | in-use    |    2 | Attached to x_db on /dev/vdb  |
+--------------------------------------+--------------+-----------+------+---------------------------------------+

And the command to attach the volume:

#openstack --debug server add volume 80bde0e8-1c17-4c77-add1-49cfb75e19f7 723dd3bc-d67a-47e7-8cb7-32ddce80f810

https://gist.github.com/solvire/ec8ad7910b138c1f8cd7c26ec9bd9ce0

The nova logs from the fuel server:

nova.compute.manager [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Attaching volume 723dd3bc-d67a-47e7-8cb7-32ddce80f810 to /dev/vdc

os_brick.initiator.connectors.iscsi [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] Multipath discovery for iSCSI enabled

nova.virt.block_device [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Driver failed to attach volume 723dd3bc-d67a-47e7-8cb7-32ddce80f810 at /dev/vdc 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Traceback (most recent call last): 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] device_type=self['device_type'], encryption=encryption) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] self._connect_volume(connection_info, disk_info) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] vol_driver.connect_volume(connection_info, disk_info) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] device_info = self.connector.connect_volume(connection_info['data']) 2016-10-21 02:00:40.285 13001 ERROR nova.virt.block_device 

nova.compute.manager [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Failed to attach 723dd3bc-d67a-47e7-8cb7-32ddce80f810 at /dev/vdc 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Traceback (most recent call last): 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4679, in _attach_volume 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] do_check_attach=False, do_driver_attach=True) 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] ret_val = method(obj, context, *args, **kwargs) 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] connector) 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] self.force_reraise() 2016-10-21 02:00:40.564 13001 ERROR nova.compute.manager [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-21 02:00:40.564 130

oslo_messaging.rpc.server [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] Exception during message handling 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-21 02:00:40.704 13001 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py",

cinder.conf

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = pkgosvg0
iscsi_protocol = iscsi
iscsi_helper = tgtadm

[LVM-backend]
volume_backend_name=LVM-backend
volumes_dir=/var/lib/cinder/volumes
iscsi_helper=tgtadm
volume_group=cinder
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
iscsi_ip_address=192.168.1.3

[cinder_netapp]
netapp_login=auser
netapp_controller_ips=
netapp_lun_ostype=linux
netapp_vserver=
netapp_server_port=
nfs_shares_config=/etc/cinder/shares.conf
netapp_sa_password=
thres_avl_size_perc_start=20
reserved_percentage=0
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_enable_multiattach=False
netapp_storage_protocol=iscsi
max_oversubscription_ratio=1.0
backend_host=str:netapp
netapp_storage_family=ontap_7mode
volume_backend_name=cinder_netapp
nfs_mount_options=
thres_avl_size_perc_stop=60
netapp_vfiler=
netapp_server_hostname=192.168.1.xx2
netapp_server_hostname=192.168.1.202
netapp_host_type=linux
expiry_thres_minutes=720
netapp_pool_name_search_pattern=(vol_openstack_junk.+)
netapp_transport_type=http
netapp_password=apswd
netapp_webservice_path=/devmgr/v2
netapp_lun_space_reservation=enabled
use_multipath_for_image_xfer=True
netapp_copyoffload_tool_path=

tgt daemon is running on both controller and compute node

service tgt status
● tgt.service - (i)SCSI target daemon
   Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-10-20 23:06:39 UTC; 3h 41min ago

From the netapp:

I removed all the initiators from the access list.

netapp01> iscsi interface accesslist show
No interface access lists

Then I tried adding them all back in so that only the proper target shows up:

From compute node:

iscsiadm -m discovery -t st -p 192.168.1.202

192.168.1.202:3260,2000 iqn.1992-08.com.netapp:sn.151741054

I can see the netapp being managed properly. (seemingly)

Fri Oct 21 01:37:02 GMT [netapp01:lun.map:info]: LUN /vol/vol_openstack_junk/volume-723dd3bc-d67a-47e7-8cb7-32ddce80f810 was mapped to initiator group openstack-509e4a19-8fb8-4e35-b276-a49bd54d5e7f=0  
Fri Oct 21 01:38:37 GMT [netapp01:lun.map.unmap:info]: LUN /vol/vol_openstack_junk/volume-723dd3bc-d67a-47e7-8cb7-32ddce80f810 unmapped from initiator group openstack-509e4a19-8fb8-4e35-b276-a49bd54d5e7f


[instance: d9b33830-7a7d-4545-bb04-a3b33d3a5d9a] Attaching volume 0de9445f-c4a5-411e-ae2f-7654bdfebd9b to /dev/vdb
Multipath discovery for iSCSI enabled
[instance: d9b33830-7a7d-4545-bb04-a3b33d3a5d9a] Driver failed to attach volume 0de9445f-c4a5-411e-ae2f-7654bdfebd9b at /dev/vdb
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
     device_type=self['device_type'], encryption=encryption)
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume
     self._connect_volume(connection_info, disk_info)
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume
     vol_driver.connect_volume(connection_info, disk_info)
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
     device_info = self.connector.connect_volume(connection_info['data'])
   File "/usr/lib/python2.7/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
     return f(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
     return f(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 396, in connect_volume
     connection_properties)
   File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 175, in _get_potential_volume_paths
     target_portal=connection_properties['target_portal'])
 TargetPortalNotFound: Unable to find target portal 10.0.0.50:3260.

38dafc55d53a4497878d5421c37e1c7c 7a4b3bb9b30e435089ca9aab12e183a7 - - -] [instance: d9b33830-7a7d-4545-bb04-a3b33d3a5d9a] Failed to attach 0de9445f-c4a5-411e-ae2f-7654bdfebd9b at /dev/vdb
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4679, in _attach_volume
    do_check_attach=False, do_driver_attach=True)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped
    ret_val = method(obj, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach
    connector)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
    device_type=self['device_type'], encryption=encryption)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume
    self._connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume
    vol_driver.connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
    device_info = self.connector.connect_volume(connection_info['data'])
  File "/usr/lib/python2.7/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 396, in connect_volume
    connection_properties)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 175, in _get_potential_volume_paths
    target_portal=connection_properties['target_portal'])
TargetPortalNotFound: Unable to find target portal 10.0.0.50:3260.

Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped
    function_name, call_dict, binary)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 66, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
    kwargs['instance'], e, sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4669, in attach_volume
    do_attach_volume(context, instance, driver_bdm)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4667, in do_attach_volume
    bdm.destroy()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4664, in do_attach_volume
    return self._attach_volume(context, instance, driver_bdm)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4687, in _attach_volume
    self.volume_api.unreserve_volume(context, bdm.volume_id)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4679, in _attach_volume
    do_check_attach=False, do_driver_attach=True)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped
    ret_val = method(obj, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach
    connector)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
    device_type=self['device_type'], encryption=encryption)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume
    self._connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume
    vol_driver.connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
    device_info = self.connector.connect_volume(connection_info['data'])
  File "/usr/lib/python2.7/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 396, in connect_volume
    connection_properties)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 175, in _get_potential_volume_paths
    target_portal=connection_properties['target_portal'])
TargetPortalNotFound: Unable to find target portal 10.0.0.50:3260.

I'm about 2 days into this one. Any help is greatly appreciated.

Edit1: reinstalled due to breaking a few things trying to mess with iscsi. Tried with and without multi-path and it still pics up the wrong portal.

Cannot attach volume - cinder + netapp + iscsi

I am able to create a volume on the NetApp SAN from horizon and the command line, but I am not able to attach that volume to any instances. I can attach a LVM volume to instances.

I'm on newton. NetApp dataontap 7-mode iscsi

The luns are created correctly so cinder can get into the netapp and configure things. I've checked cinder.conf for the ordinary things listed in other posts.

openstack server list | grep x_db
| 80bde0e8-1c17-4c77-add1-49cfb75e19f7 | x_db    | ACTIVE | admin_internal_net=192.168.111.9                 |

openstack volume list
+--------------------------------------+--------------+-----------+------+---------------------------------------+
| ID                                   | Display Name | Status    | Size | Attached to                           |
+--------------------------------------+--------------+-----------+------+---------------------------------------+
| 723dd3bc-d67a-47e7-8cb7-32ddce80f810 | med_test     | available |    3 |                                       |
| dd76afad-ff33-4379-ab48-342c0c8674b0 | test_vol     | in-use    |    2 | Attached to x_db on /dev/vdb  |
+--------------------------------------+--------------+-----------+------+---------------------------------------+

And the command to attach the volume:

#openstack --debug server add volume 80bde0e8-1c17-4c77-add1-49cfb75e19f7 723dd3bc-d67a-47e7-8cb7-32ddce80f810

https://gist.github.com/solvire/ec8ad7910b138c1f8cd7c26ec9bd9ce0

The nova logs from the fuel server:

nova.compute.manager [req-5e728c4a-49d0-4175-b3ce-b3aaaa31d2a9 db2fd54547014d489942c63f96fcf315 64f7b2e6f00a4bcd9c896fd3fba7ed33 - - -] [instance: 80bde0e8-1c17-4c77-add1-49cfb75e19f7] Attaching volume 723dd3bc-d67a-47e7-8cb7-32ddce80f810 to /dev/vdc

cinder.conf

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = pkgosvg0
iscsi_protocol = iscsi
iscsi_helper = tgtadm

[LVM-backend]
volume_backend_name=LVM-backend
volumes_dir=/var/lib/cinder/volumes
iscsi_helper=tgtadm
volume_group=cinder
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
iscsi_ip_address=192.168.1.3

[cinder_netapp]
netapp_login=auser
netapp_controller_ips=
netapp_lun_ostype=linux
netapp_vserver=
netapp_server_port=
nfs_shares_config=/etc/cinder/shares.conf
netapp_sa_password=
thres_avl_size_perc_start=20
reserved_percentage=0
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_enable_multiattach=False
netapp_storage_protocol=iscsi
max_oversubscription_ratio=1.0
backend_host=str:netapp
netapp_storage_family=ontap_7mode
volume_backend_name=cinder_netapp
nfs_mount_options=
thres_avl_size_perc_stop=60
netapp_vfiler=
netapp_server_hostname=192.168.1.202
netapp_host_type=linux
expiry_thres_minutes=720
netapp_pool_name_search_pattern=(vol_openstack_junk.+)
netapp_transport_type=http
netapp_password=apswd
netapp_webservice_path=/devmgr/v2
netapp_lun_space_reservation=enabled
use_multipath_for_image_xfer=True
netapp_copyoffload_tool_path=

tgt daemon is running on both controller and compute node

service tgt status
● tgt.service - (i)SCSI target daemon
   Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-10-20 23:06:39 UTC; 3h 41min ago

From the netapp:

I removed all the initiators from the access list.

netapp01> iscsi interface accesslist show
No interface access lists

Then I tried adding them all back in so that only the proper target shows up:

From compute node:

# iscsiadm -m discovery -t st -p 192.168.1.202

192.168.1.202 192.168.1.202:3260,2000 iqn.1992-08.com.netapp:sn.151741054

iqn.1992-08.com.netapp:sn.151741054

I can see the netapp being managed properly. (seemingly)

Fri Oct 21 01:37:02 GMT [netapp01:lun.map:info]: LUN /vol/vol_openstack_junk/volume-723dd3bc-d67a-47e7-8cb7-32ddce80f810 was mapped to initiator group openstack-509e4a19-8fb8-4e35-b276-a49bd54d5e7f=0  
Fri Oct 21 01:38:37 GMT [netapp01:lun.map.unmap:info]: LUN /vol/vol_openstack_junk/volume-723dd3bc-d67a-47e7-8cb7-32ddce80f810 unmapped from initiator group openstack-509e4a19-8fb8-4e35-b276-a49bd54d5e7f


[instance: d9b33830-7a7d-4545-bb04-a3b33d3a5d9a] Attaching volume 0de9445f-c4a5-411e-ae2f-7654bdfebd9b to /dev/vdb
Multipath discovery for iSCSI enabled
[instance: d9b33830-7a7d-4545-bb04-a3b33d3a5d9a] Driver failed to attach volume 0de9445f-c4a5-411e-ae2f-7654bdfebd9b at /dev/vdb
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
     device_type=self['device_type'], encryption=encryption)
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume
     self._connect_volume(connection_info, disk_info)
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume
     vol_driver.connect_volume(connection_info, disk_info)
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
     device_info = self.connector.connect_volume(connection_info['data'])
   File "/usr/lib/python2.7/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
     return f(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
     return f(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 396, in connect_volume
     connection_properties)
   File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 175, in _get_potential_volume_paths
     target_portal=connection_properties['target_portal'])
 TargetPortalNotFound: Unable to find target portal 10.0.0.50:3260.

38dafc55d53a4497878d5421c37e1c7c 7a4b3bb9b30e435089ca9aab12e183a7 - - -] [instance: d9b33830-7a7d-4545-bb04-a3b33d3a5d9a] Failed to attach 0de9445f-c4a5-411e-ae2f-7654bdfebd9b at /dev/vdb
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4679, in _attach_volume
    do_check_attach=False, do_driver_attach=True)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped
    ret_val = method(obj, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach
    connector)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
    device_type=self['device_type'], encryption=encryption)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume
    self._connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume
    vol_driver.connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
    device_info = self.connector.connect_volume(connection_info['data'])
  File "/usr/lib/python2.7/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 396, in connect_volume
    connection_properties)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 175, in _get_potential_volume_paths
    target_portal=connection_properties['target_portal'])
TargetPortalNotFound: Unable to find target portal 10.0.0.50:3260.

Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped
    function_name, call_dict, binary)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 66, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
    kwargs['instance'], e, sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4669, in attach_volume
    do_attach_volume(context, instance, driver_bdm)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4667, in do_attach_volume
    bdm.destroy()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4664, in do_attach_volume
    return self._attach_volume(context, instance, driver_bdm)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4687, in _attach_volume
    self.volume_api.unreserve_volume(context, bdm.volume_id)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4679, in _attach_volume
    do_check_attach=False, do_driver_attach=True)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped
    ret_val = method(obj, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach
    connector)
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
    device_type=self['device_type'], encryption=encryption)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1131, in attach_volume
    self._connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1072, in _connect_volume
    vol_driver.connect_volume(connection_info, disk_info)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
    device_info = self.connector.connect_volume(connection_info['data'])
  File "/usr/lib/python2.7/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 396, in connect_volume
    connection_properties)
  File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connectors/iscsi.py", line 175, in _get_potential_volume_paths
    target_portal=connection_properties['target_portal'])
TargetPortalNotFound: Unable to find target portal 10.0.0.50:3260.

I'm about 2 days into this one. Any help is greatly appreciated.

Edit1: reinstalled due to breaking a few things trying to mess with iscsi. Tried with and without multi-path and it still pics up the wrong portal.