Ask Your Question
0

ISCSI failure after restart of compute/cinder

asked 2015-01-23 07:04:20 -0500

trigger gravatar image

updated 2015-01-23 08:13:24 -0500

Hello,

  • CentOS 6.5
  • OpenStack Icehouse

We are running compute hosts that acts as both Nova Compute and Cinder Volume nodes. They get ISCSI from our SAN for shared storage so that we can create Cinder volumes on there.

After a compute node is restarted (running both compute and cinder) ISCSI fails with the following logs, please see http://pastebin.com/mM3UbaXB

  • ISCSI volume not yet found at: vda.
  • NovaException: iSCSI device not found at

What do I need to do in order for Cinder to setup the appropriate ISCSI connections again? I can see it when checking "iscsiadm -m session" tcp: [33] 192.168.100.10:3260,1 iqn.2010-10.org.openstack:volume-5d51f748-aec8-4da7-a3d2-1083afb720b6

I can't see the volume when running "fdisk -l" so the ISCSI is not working properly as it's not seeing the logical volume. Anybody have any suggestions for me on how to resolved this?

I have tried rebooting it several times again, also restarting tgtd, libvirtd etc. I have got it working once, but then after restart it didn't work again and I can't remember why it started working, "it just did".

Best regards

edit retag flag offensive close merge delete
add a comment

2 answers

Sort by ยป oldest newest most voted
1

answered 2015-01-27 09:56:36 -0500

trigger gravatar image

updated 2015-01-27 09:57:58 -0500

Issue was that tgtadm failed to create the ISCSI target, the volume entered 'error' state and Cinder does not try to create targets or "fix" volumes that is in the state 'error'. The issue can be seen in the volume.log cinder log as:

2015-01-23 10:53:15.791 3193 ERROR cinder.brick.iscsi.iscsi [req-c3c06a94-c49e-470b-a956-cfc0802024bd - - - - -] Failed to recover attempt to create iscsi backing lun for volume id:iqn.2010-10.org.openstack:volume-5d51f748-aec8-4da7-a3d2-1083afb720b6: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/cinder-volumes/volume-5d51f748-aec8-4da7-a3d2-1083afb720b6
2015-01-23 10:53:15.871 3193 ERROR cinder.volume.manager [req-c3c06a94-c49e-470b-a956-cfc0802024bd - - - - -] Failed to re-export volume 5d51f748-aec8-4da7-a3d2-1083afb720b6: setting to error state
2015-01-23 10:53:15.872 3193 ERROR cinder.volume.manager [req-c3c06a94-c49e-470b-a956-cfc0802024bd - - - - -] Failed to create iscsi target for volume volume-5d51f748-aec8-4da7-a3d2-1083afb720b6.
2015-01-23 10:53:15.872 3193 TRACE cinder.volume.manager ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-5d51f748-aec8-4da7-a3d2-1083afb720b6.

A bug fix is already proposed here: https://review.openstack.org/#/c/147566/5 (https://review.openstack.org/#/c/1475...)

To solve the issue I did the following:

cinder reset-state --state in-use 5d51f748-aec8-4da7-a3d2-1083afb720b6
/etc/init.d/openstack-cinder-volume restart
iscsiadm -m node -T iqn.2010-10.org.openstack:volume-5d51f748-aec8-4da7-a3d2-1083afb720b6 -l

Thanks to Avishay Traeger (avishay on #openstack-cinder) for the help.

edit flag offensive delete link more
add a comment
0

answered 2015-01-23 07:51:27 -0500

dbaxps gravatar image

updated 2015-01-23 07:59:45 -0500

Please, view
1. OpenStack Documentation Configure a Block Storage Service controller : http://docs.openstack.org/havana/inst...
2. OpenStack Documentation Configure a Block Storage Service node : http://docs.openstack.org/havana/inst...

iSCSI Target host is supposed to run service target && utlity targetcli on modern systems (CentOS 7, F21)
iSCSI Initiator host is supposed to run service iscsid to detect LVMiSCSI targets , located on Storage Node - iSCSI Target Host, iSCSI initiator host is your Compute Node (Nodes)
A sample of tuning simple TwoNode Cluster I posted here http://lxer.com/module/newswire/view/... for F21 and here http://lxer.com/module/newswire/view/... for CentOS 7
iSCSI Target Host is the node (storage node) hosting your cinder back end LVMs
Cinder configuration is explained in details in both posts mentioned above, it also differs very much on modern systems running Juno from IceHouse

edit flag offensive delete link more

Comments

Hello,

Thanks for the effort, however it's not even close too what I'm seeking help with. I also forgot to mention I'm running CentOS 6.5 with OpenStack Icehouse.

trigger gravatar imagetrigger ( 2015-01-23 08:13:04 -0500 )edit

In your case tgtd will work on iSCSI Target. Can you install targetcli on CentOS 6.5 ?
If yes , it won't be any difference. If no then should be tool to create iSCSI target configuration on 6.5.

dbaxps gravatar imagedbaxps ( 2015-01-23 08:52:00 -0500 )edit

I'm running tgtd with "include /etc/cinder/volumes/*" in my /etc/tgt/targets.conf configuration. And in /etc/cinder/cinder.conf I got "iscsi_helper=tgtadm", I'm using tgtd. For example, please see http://pastebin.com/EvxfM62N for additional information.

trigger gravatar imagetrigger ( 2015-01-23 09:34:25 -0500 )edit

Disregarding core issue (IceHouse,Cinder), which system tool should you use on iSCSI Target server to create iSCSI target ( no connection to Openstack ) ?

dbaxps gravatar imagedbaxps ( 2015-01-23 10:46:26 -0500 )edit

Ok. I know. What says ?

# tgtadm --lld iscsi --op show --mode target
dbaxps gravatar imagedbaxps ( 2015-01-23 11:03:08 -0500 )edit
see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2015-01-23 07:04:20 -0500

Seen: 5,351 times

Last updated: Jan 27 '15

Related questions

how to debug kolla cinder lvm2 with iscsi

nova detach iscsi error

Volume did not finish being created even

How to configure openstack (nova and cinder) to use LUNs from SAN ?

Cinder provider location address

cinder support for GFS2 backend on SAN

Creating an instance on the second compute node

Can nova-compute-lxd run lxd instances alongside kvm instances

Image stuck in spawning state

cinder seems to timeout on large volume creation?