Ask Your Question
0

cinder broken iscsiadm: No session found.

asked 2013-12-17 04:29:10 -0500

Sifty gravatar image

updated 2013-12-17 08:48:57 -0500

A power outage caused our openstack servers to die.

Most things recovered however cinder volumes are not attachable and just error.

 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a -p 172.25.0.14:3260 --rescan
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Exit code: 21
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stdout: ''
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stderr: 'iscsiadm: No session found.\n'
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp

get the folliowing on controller when restarting iscsi...

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

and on compute node

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-672e1a41-07f4-4260-8831-df153a952ed6, portal: 172.18.6.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

So assume there is something wrong with the way the targets are being published on the controller.

Initial issue was the cinder-volumes vg was inactive fixed with vgchange -ay restarted tgtd and cinder services expecting all to be ok

on compute node.. shows only one available volume.

 iscsiadm -m discovery -t sendtargets -p <cinder server>
 <cinder server>:3260,1 iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598

on cinder host... there are 5 defined.

tgtadm --lld iscsi --mode target --op show

shows all targets for all 5 volumes defined in cinder server.

 Target 1: iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598
  System information:
      Driver: iscsi
      State: ready
  I_T nexus information:
  LUN information:
      LUN: 0
        Type: controller
        SCSI ID: IET     00010000
        SCSI SN: beaf10
        Size: 0 MB, Block size: 1
        Online: Yes
        Removable media: No
        Prevent removal: No
        Readonly: No
        Backing store type: null
        Backing store path: None
        Backing store flags:
    LUN: 1
        Type: disk
        SCSI ID: IET     00010001
        SCSI SN: beaf11
        Size: 34360 MB, Block size: 512
        Online: Yes
        Removable media: No
        Prevent removal: No
        Readonly: No
        Backing store type: rdwr
        Backing store path: /dev/cinder-volumes/volume-03baa6b7-3601-4b6f-a842-5c634c627598
        Backing store flags:
  Account information:
  ACL information:
      ALL
 Target 2: iqn.2010-10.org.openstack:volume-301d9c26-352d-471b-a757-96a8ffcae742
    System information:
 ....
edit retag flag offensive close merge delete

Comments

What does "iscsiadm -m discovery -t sendtargets -p <cinder-server>" show when run on your compute nodes? How about "tgtadm --lld iscsi --mode target --op show" when run on your Cinder host?

larsks gravatar imagelarsks ( 2013-12-17 08:36:53 -0500 )edit

iscsiadm -m discovery -t sendtargets -p <cinder-server> Shows only 1 target when there are 5 volumes defined in cinder <cinder-server>:3260,1 iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598 On the cinder server tgtadm --lld iscsi --mode target --op show" shows all 5 volumes

Sifty gravatar imageSifty ( 2013-12-17 08:43:34 -0500 )edit

Just wondering if this could be a stop all services and stop iscsid services... start them again to get the iscsid to publish correctly ?

Sifty gravatar imageSifty ( 2013-12-17 08:56:01 -0500 )edit

OK still no luck with this... when the system died the cinder volumes were attached no matter what i do I cannot detach the volumes from the shutdown instances - is there a way to manually detach the volumes using mysql?

Sifty gravatar imageSifty ( 2013-12-17 16:14:57 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2013-12-19 11:06:16 -0500

Sifty gravatar image

updated 2013-12-19 11:07:42 -0500

Ok fixed.... and fickle... but this is a good tip if anyone has this...

Down to fact i use iscsi for storage san backend on same blade as controller node.

You can double check if there is a problem by

lsof -i -n | grep iscsi

iscsid     1462      root    9u  IPv4 1591345      0t0  TCP 10.0.10.14:50846->10.0.10.66:iscsi-target (ESTABLISHED)
iscsid     1462      root   11u  IPv4 1591350      0t0  TCP 10.0.20.14:51876->10.0.20.65:iscsi-target (ESTABLISHED)
iscsid     1462      root   13u  IPv4 1591353      0t0  TCP 10.0.10.14:54707->10.0.10.64:iscsi-target (ESTABLISHED)
iscsid     1462      root   13u  IPv4 1591353      0t0  TCP 172.25.0.14:54707->172.25.0.14:iscsi-target (ESTABLISHED)
tgtd      25585      root    4u  IPv4 2078033      0t0  TCP *:iscsi-target (LISTEN)
tgtd      25585      root    5u  IPv6 2078034      0t0  TCP *:iscsi-target (LISTEN)
tgtd      25588      root    4u  IPv4 2078033      0t0  TCP *:iscsi-target (LISTEN)
tgtd      25588      root    5u  IPv6 2078034      0t0  TCP *:iscsi-target (LISTEN)

see what is running. this daemon should NOT have been connected...

  iscsid     1462      root   13u  IPv4 1591353      0t0  TCP 172.25.0.14:54707->172.25.0.14:iscsi-target (ESTABLISHED)

I fixed by stopping tgtd and killing any rogue tgtd threads still running

edit flag offensive delete link more

Comments

Thanks Sifty! This was a great help after a recent outage we had in our setup. Been breaking my head for few hours now :)

vijesh shetty gravatar imagevijesh shetty ( 2014-06-09 08:07:11 -0500 )edit

Can you elaborate on the last steps of your fix? Surely you didn't just leave tgtd stopped, right?

sparr gravatar imagesparr ( 2015-01-26 14:23:56 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2013-12-17 04:29:10 -0500

Seen: 6,000 times

Last updated: Dec 19 '13