issue while detaching volume (FC type) from instance

asked 2019-07-15 04:03:35 -0500

aviv-opens gravatar image

Hey all, lately I added a second tier for my cinder backend, an FC storage type. all the setup goes well the service was up and I was able to execute any task related to the capability of the driver. just except detach and apparently it also affected when the instance goes for a reboot. the same main error that shown

"Stdout: u'Jul 15 02:46:50 | /dev/disk/by-id/dm-uuid-mpath-20024f400554505a0: map in use\nJul 15 02:46:50 | failed to remove multipath map /dev/disk/by-id/dm-uuid-mpath-20024f400554505a0\n' Stderr: u'': ProcessExecutionError: Unexpected error while running command. 2019-07-15 02:47:30.904 8111 INFO nova.compute.manager [req-28dce3d5-3412-412e-9bc7-a0be5e921bf9 3a67a05137e5a2e48c975dab3ed0d493c890ae27f5f343f567a59a487a465ce5 7d91af2e9c8c4dc392f61437ad932ba2 - d70a4991744248d9a6733356e668dfc6 d70a4991744248d9a6733356e668dfc6] [instance: 1649960c-dbff-4776-b6a5-c0719af2a130] Successfully reverted task state from reboot_started_hard on failure for instance. 2019-07-15 02:47:30.947 8111 ERROR oslo_messaging.rpc.server [req-28dce3d5-3412-412e-9bc7-a0be5e921bf9 3a67a05137e5a2e48c975dab3ed0d493c890ae27f5f343f567a59a487a465ce5 7d91af2e9c8c4dc392f61437ad932ba2 - d70a4991744248d9a6733356e668dfc6 d70a4991744248d9a6733356e668dfc6] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: multipath -f /dev/disk/by-id/dm-uuid-mpath-20024f400554505a0 "

as a workaround, I running "echo 1 > /sys/bus/scsi/drivers/sd/$DEVICE/delete" and then seems he succeeded to run after "multipath -f"

Anyone also interrupts in this issue and can help me here? Thanks.

I running queens 17.1.9 deployed by openstack-ansible

edit retag flag offensive close merge delete

Comments

I would run the multipath command manually on the compute node, also with high levels of verbosity, and look for errors on stdout or among the kernel messages.

Bernd Bausch gravatar imageBernd Bausch ( 2019-07-16 08:50:07 -0500 )edit

Hi Bernd, I didn't find anything suspicious on the log, https://pastebin.com/Dhc0HtRS (attach here).

aviv-opens gravatar imageaviv-opens ( 2019-07-21 10:19:41 -0500 )edit

seems the disk was exported to host are not clean up well. the map of the disk still exists on the host even if he was deleted so when creating a new one he got the same WWID that was assigned to the old volume and from the host side he still in a map, so each task to remove him failed.

aviv-opens gravatar imageaviv-opens ( 2019-07-21 10:20:38 -0500 )edit