0

Live migration "volume backed" no error, but no change

asked 2016-12-28 20:47:20 -0600

xdaxdb
57 ●7 ●9 ●16

updated 2016-12-29 21:31:24 -0600

I am trying to live migrate an instance which is running from a cinder volume. The command completes without any errors, and I can't find any errors in logs on the controller, source host, or destination host. Dashboard/CLI (nova and openstack commands) same result.

The only thing I can find that is remotely suspicious is in nova-compute.log on the destination host:

WARNING os_brick.initiator.connectors.iscsi ... ISCSI volume not yet found at: [u'...']. Will rescan & retry.  Try number: 0.
WARNING nova.compute.manager ... Detaching volume from unknown instance
WARNING nova.virt.libvirt.driver ... During detach_volume, instance disappeared.

There are some messages in syslog that correspond with this, but I'm not certain if there is a problem or if this is normal behavior when the host attaches a new volume and re-scans the SCSI bus:

kernel: sd 11:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
kernel: sd 11:0:0:3: [sdi] Unit Not Ready
kernel: sd 11:0:0:3: [sdi] Sense Key : Illegal Request [current]
kernel: sd 11:0:0:3: [sdi] Add. Sense: Logical unit not supported
kernel: sd 11:0:0:3: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
kernel: sd 11:0:0:3: [sdi] Sense Key : Illegal Request [current]
kernel: sd 11:0:0:3: [sdi] Add. Sense: Logical unit not supported
kernel: sd 11:0:0:3: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
kernel: sd 11:0:0:3: [sdi] Sense Key : Illegal Request [current]
kernel: sd 11:0:0:3: [sdi] Add. Sense: Logical unit not supported
kernel: scsi 11:0:0:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
kernel: scsi 11:0:0:1: alua: supports implicit TPGS
kernel: scsi 11:0:0:1: alua: port group 00 rel port 180
kernel: scsi 11:0:0:1: alua: rtpg failed with 8000002
kernel: scsi 11:0:0:1: alua: port group 00 state A non-preferred supports tolusna
kernel: scsi 11:0:0:1: alua: Attached
kernel: sd 11:0:0:1: Attached scsi generic sg3 type 0
kernel: sd 11:0:0:1: [sdd] 209715200 512-byte logical blocks: (107 GB/100 GiB)
kernel: sd 11:0:0:1: [sdd] Write Protect is off
kernel: sd 11:0:0:1: [sdd] Mode Sense: 97 00 10 08
kernel: sd 11:0:0:1: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
kernel:  sdd: sdd1

My only theory is that the timing of the process is a little off and Nova isn't seeing the volume available on the destination host and is aborting the migration. But there is no explicit error message.

edit retag flag offensive close merge delete

Comments

Any thoughts anyone might have about what makes a volume backed migration different than a typical image backed migration would be helpful.

xdaxdb ( 2016-12-31 22:05:22 -0600 )edit

2 answers

Sort by » oldest newest most voted

0

answered 2016-12-28 23:42:27 -0600

silently
31 ●2 ●2

have a look at cinder log file ?

edit flag offensive delete link

Comments

There's nothing very interesting in there, basically:

Volume info retrieved successfully.
Initialize volume connection completed successfully.
Volume info retrieved successfully.
Terminate volume connection completed successfully.
Unreserve volume completed successfully.

xdaxdb ( 2016-12-29 01:39:09 -0600 )edit

0

answered 2017-01-13 16:28:05 -0600

xdaxdb
57 ●7 ●9 ●16

The cause of this had nothing to do with Cinder or Volume Backed Instances.

The cause was the default libvirtd settings. The libvirtd settings on RH/CentOS by default do not listen for external connections. In order for Live migration to work the hypervisors need to talk to each other directly.

Uncomment LIBVIRTD_ARGS="--listen" in /etc/sysconfig/libvirtd on the compute nodes.

I'm not sure if Nova is supposed to manage this or not (it should). I've noticed that the TLS settings in libvirtd are ignored by Nova, and there are settings in nova.conf that control TLS (which I am still struggling with, I don't think they ware working properly)

edit flag offensive delete link

Comments

Which version of Nova? 13.0.0 Mitaka, 14.0.0 Liberty, other? If you're on Newton what's the value of the live_migration_tunnelled config option in nova.conf on the compute nodes?

mriedem ( 2017-01-14 09:36:37 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2016-12-28 20:47:20 -0600

Seen: 1,292 times

Last updated: Jan 13 '17

Related questions