Revision history [back]

click to hide/show revision 1
initial version

cinder broken iscsiadm: No session found.

Power outage caused our openstack servers to die... i know not ideal.

Most things recovered however cinder volumes are not attachable and just error.

 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a -p 172.25.0.14:3260 --rescan
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Exit code: 21
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stdout: ''
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stderr: 'iscsiadm: No session found.\n'
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp

get the folliowing on controller when restarting iscsi...

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

and on compute node

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-672e1a41-07f4-4260-8831-df153a952ed6, portal: 172.18.6.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

So assume there is something wrong with the way the targets are being published on the controller.

Initial issue was the cinder-volumes vg was inactive fixed with vgchange -ay restarted tgtd and cinder services expecting all to be ok

but have now hit a road block... and confidence in OS taken a massive hit.

click to hide/show revision 2
No.2 Revision

cinder broken iscsiadm: No session found.

Power A power outage caused our openstack servers to die... i know not ideal.die.

Most things recovered however cinder volumes are not attachable and just error.

 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a -p 172.25.0.14:3260 --rescan
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Exit code: 21
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stdout: ''
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stderr: 'iscsiadm: No session found.\n'
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp

get the folliowing on controller when restarting iscsi...

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

and on compute node

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-672e1a41-07f4-4260-8831-df153a952ed6, portal: 172.18.6.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

So assume there is something wrong with the way the targets are being published on the controller.

Initial issue was the cinder-volumes vg was inactive fixed with vgchange -ay restarted tgtd and cinder services expecting all to be ok

but have now hit a road block... and confidence in OS taken a massive hit.

cinder broken iscsiadm: No session found.

A power outage caused our openstack servers to die.

Most things recovered however cinder volumes are not attachable and just error.

 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a -p 172.25.0.14:3260 --rescan
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Exit code: 21
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stdout: ''
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stderr: 'iscsiadm: No session found.\n'
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp

get the folliowing on controller when restarting iscsi...

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

and on compute node

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-672e1a41-07f4-4260-8831-df153a952ed6, portal: 172.18.6.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

So assume there is something wrong with the way the targets are being published on the controller.

Initial issue was the cinder-volumes vg was inactive fixed with vgchange -ay restarted tgtd and cinder services expecting all to be ok

on compute node..

 iscsiadm -m discovery -t sendtargets -p <cinder server>
 <cinder server>:3260,1 iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598

on cinder host...

tgtadm --lld iscsi --mode target --op show

shows all targets for all 5 volumes defined in cinder server.

 Target 1: iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598
  System information:
      Driver: iscsi
      State: ready
  I_T nexus information:
  LUN information:
      LUN: 0
        Type: controller
        SCSI ID: IET     00010000
        SCSI SN: beaf10
        Size: 0 MB, Block size: 1
        Online: Yes
        Removable media: No
        Prevent removal: No
        Readonly: No
        Backing store type: null
        Backing store path: None
        Backing store flags:
    LUN: 1
        Type: disk
        SCSI ID: IET     00010001
        SCSI SN: beaf11
        Size: 34360 MB, Block size: 512
        Online: Yes
        Removable media: No
        Prevent removal: No
        Readonly: No
        Backing store type: rdwr
        Backing store path: /dev/cinder-volumes/volume-03baa6b7-3601-4b6f-a842-5c634c627598
        Backing store flags:
  Account information:
  ACL information:
      ALL
 Target 2: iqn.2010-10.org.openstack:volume-301d9c26-352d-471b-a757-96a8ffcae742
    System information:
 ....

cinder broken iscsiadm: No session found.

A power outage caused our openstack servers to die.

Most things recovered however cinder volumes are not attachable and just error.

 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a -p 172.25.0.14:3260 --rescan
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Exit code: 21
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stdout: ''
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp Stderr: 'iscsiadm: No session found.\n'
 2013-12-17 10:04:21.870 4283 TRACE nova.openstack.common.rpc.amqp

get the folliowing on controller when restarting iscsi...

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

and on compute node

Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-9dc5ac4e-8851-4b20-8b5c-ee3c0bd1a14a, portal: 172.25.0.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-672e1a41-07f4-4260-8831-df153a952ed6, portal: 172.18.6.14,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
                                                       [  OK  ]

So assume there is something wrong with the way the targets are being published on the controller.

Initial issue was the cinder-volumes vg was inactive fixed with vgchange -ay restarted tgtd and cinder services expecting all to be ok

on compute node..node.. shows only one available volume.

 iscsiadm -m discovery -t sendtargets -p <cinder server>
 <cinder server>:3260,1 iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598

on cinder host...host... there are 5 defined.

tgtadm --lld iscsi --mode target --op show

shows all targets for all 5 volumes defined in cinder server.

 Target 1: iqn.2010-10.org.openstack:volume-03baa6b7-3601-4b6f-a842-5c634c627598
  System information:
      Driver: iscsi
      State: ready
  I_T nexus information:
  LUN information:
      LUN: 0
        Type: controller
        SCSI ID: IET     00010000
        SCSI SN: beaf10
        Size: 0 MB, Block size: 1
        Online: Yes
        Removable media: No
        Prevent removal: No
        Readonly: No
        Backing store type: null
        Backing store path: None
        Backing store flags:
    LUN: 1
        Type: disk
        SCSI ID: IET     00010001
        SCSI SN: beaf11
        Size: 34360 MB, Block size: 512
        Online: Yes
        Removable media: No
        Prevent removal: No
        Readonly: No
        Backing store type: rdwr
        Backing store path: /dev/cinder-volumes/volume-03baa6b7-3601-4b6f-a842-5c634c627598
        Backing store flags:
  Account information:
  ACL information:
      ALL
 Target 2: iqn.2010-10.org.openstack:volume-301d9c26-352d-471b-a757-96a8ffcae742
    System information:
 ....