Ask Your Question
0

Why is cinder volume migration failing?

asked 2015-11-04 09:27:27 -0600

bnordgren gravatar image

updated 2015-11-04 09:29:52 -0600

The context is that I have an RDO openstack Kilo cluster running on Centos 7 composed of one controller and five compute nodes. Each of the compute nodes is also running cinder-volume and manages ~2Tb of space. One node has a disk which is behaving as if failure is imminent. I am trying to migrate volumes off of this node to other nodes.

There is absolutely no output from the cinder migrate command.

The volume.log on the destination node contains the following. The top hit when googling the warning message is the python source code. I assume that means this is not a showstopper, but in any case google did not return anything I understood as a description of the problem or indicators of possible solutions. As far as I can tell, cinder takes about 20 seconds to change its mind from "creating a new volume on the target" to "deleting". I don't understand the reason for the change. Can you help me understand whether the warning is relevant or really what the first step in debugging this problem is? 

2015-11-03 16:45:35.614 2259 INFO cinder.volume.flows.manager.create_volume [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Volume ead174e4-e7d0-45e9-804d-47b915a48492: being created as raw with specification: {'status': u'creating', 'volume_size': 200, 'volume_name': u'volume-ead174e4-e7d0-45e9-804d-47b915a48492'}
2015-11-03 16:45:36.090 2259 INFO cinder.volume.flows.manager.create_volume [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Volume volume-ead174e4-e7d0-45e9-804d-47b915a48492 (ead174e4-e7d0-45e9-804d-47b915a48492): created successfully
2015-11-03 16:45:43.154 2259 INFO cinder.volume.targets.lio [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Creating iscsi_target for volume: volume-ead174e4-e7d0-45e9-804d-47b915a48492
2015-11-03 16:45:43.611 2259 WARNING cinder.volume.targets.lio [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Failed to save iscsi LIO configuration when modifying volume id: volume-ead174e4-e7d0-45e9-804d-47b915a48492.
2015-11-03 16:45:43.963 2259 WARNING cinder.volume.targets.lio [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Failed to save iscsi LIO configuration when modifying volume id: ead174e4-e7d0-45e9-804d-47b915a48492.
2015-11-03 16:46:07.817 2259 WARNING cinder.volume.targets.lio [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Failed to save iscsi LIO configuration when modifying volume id: ead174e4-e7d0-45e9-804d-47b915a48492.
2015-11-03 16:46:08.087 2259 INFO cinder.volume.manager [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] volume ead174e4-e7d0-45e9-804d-47b915a48492: deleting
2015-11-03 16:46:08.234 2259 INFO cinder.volume.targets.lio [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Removing iscsi_target: ead174e4-e7d0-45e9-804d-47b915a48492
2015-11-03 16:46:08.537 2259 WARNING cinder.volume.targets.lio [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Failed to save iscsi LIO configuration when modifying volume id: ead174e4-e7d0-45e9-804d-47b915a48492.
2015-11-03 16:46:08.733 2259 INFO cinder.volume.utils [req-9407ea97-a0a3-4c6d-b72c-4f0edbf19d75 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Performing secure delete on volume: /dev/mapper/cinder--volumes-volume--ead174e4--e7d0--45e9--804d--47b915a48492
edit retag flag offensive close merge delete
add a comment

2 answers

Sort by ยป oldest newest most voted
1

answered 2015-11-09 19:07:30 -0600

bnordgren gravatar image

Debugging this for a few days now, I noticed the following in /var/log/messages on the node hosting the volume to be migrated. (The following excerpt is from a different attempt, obviously.) The symlink for the volume to be migrated was not appearing in /dev/disk/by-path. I flailed around for a couple of days trying to figure out how to make the symlink appear, to no avail. Rebooting the node hosting the volume to be migrated, though a blunt instrument, cleared the problem up. I still do not know the root of the problem. If you can explain how to debug/fix this problem without rebooting, you get your answer accepted. Until then, this is all I got.

Nov  4 20:23:13 compute-0-1 iscsid: conn 0 login rejected: initiator failed authorization with target
Nov  4 20:23:15 compute-0-1 cinder-volume: 2015-11-04 20:23:15.105 6738 WARNING cinder.brick.initiator.connector [req-65c3e6b2-2fec-4464-94a1-8695add346d5 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-10.0.2.11:3260-iscsi-iqn.2010-10.org.openstack:volume-cd73a394-28b9-4cac-a027-56c8362954ff-lun-0']. Will rescan & retry.  Try number: 0
Nov  4 20:23:16 compute-0-1 cinder-volume: 2015-11-04 20:23:16.268 6738 WARNING cinder.brick.initiator.connector [req-65c3e6b2-2fec-4464-94a1-8695add346d5 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-10.0.2.11:3260-iscsi-iqn.2010-10.org.openstack:volume-cd73a394-28b9-4cac-a027-56c8362954ff-lun-0']. Will rescan & retry.  Try number: 1
Nov  4 20:23:16 compute-0-1 iscsid: connection4:0 is operational after recovery (9 attempts)
Nov  4 20:23:17 compute-0-1 cinder-volume: 2015-11-04 20:23:17.188 6738 INFO cinder.volume.manager [req-d149e71e-3f28-41a6-bd3b-08b56545e005 - - - - -] Updating volume status
Nov  4 20:23:20 compute-0-1 cinder-volume: 2015-11-04 20:23:20.433 6738 WARNING cinder.brick.initiator.connector [req-65c3e6b2-2fec-4464-94a1-8695add346d5 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-10.0.2.11:3260-iscsi-iqn.2010-10.org.openstack:volume-cd73a394-28b9-4cac-a027-56c8362954ff-lun-0']. Will rescan & retry.  Try number: 2
Nov  4 20:23:29 compute-0-1 cinder-volume: 2015-11-04 20:23:29.595 6738 ERROR cinder.volume.driver [req-65c3e6b2-2fec-4464-94a1-8695add346d5 530392db20334fb49460c40facf9f6b1 0ed69c3d01094c18b8c0eea04060e916 - - -] Failed to attach volume cd73a394-28b9-4cac-a027-56c8362954ff
edit flag offensive delete link more
add a comment
0

answered 2015-11-09 21:52:29 -0600

shashidhar gravatar image

I feel this may happen because of python version mis-match. Did you try resolving this by replacing python version 2.6 or older with latest python 2.7 0r python 3.4.2.

edit flag offensive delete link more
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

subscribe to rss feed

Stats

Asked: 2015-11-04 09:27:27 -0600

Seen: 762 times

Last updated: Nov 09 '15

Related questions

Huawei OceanStor Cinder drivers in RDO

how to debug kolla cinder lvm2 with iscsi

cannot see cinder-volume in cinder service-list

Cinder + LVM: No valid host found

Ceilometer - No meters / stats ? [closed]

cinder, attached volumes migration error

Cinder Unable to connect to AMQP server

Networking for lab built with RDO allinone

How to rightly configure OpenLDAP for Openstack Kilo

cinder in two node setup [closed]