Ask Your Question
0

"nova start" fails to find cinder volume, yet volume is attached to node

asked 2017-04-10 15:20:12 -0500

jkilborn gravatar image

updated 2017-04-10 15:29:17 -0500

Upgraded from liberty to newton. Trying to start vm back up on upgraded compute node. The vm was shutdown, then the update was performed, now I can't start that vm. I can create a new vm, cinder backed, and it creates the volume on the dell san , and attaches to host, and starts fine. However, the existing VM keeps failing to start.

The device in the libvirt.xml shows the disk dev as :

source dev="/dev/disk/by-id/dm-uuid-mpath-36000d3100314f600000000000000018a"/

and multipath -l shows the device as well

mpathcd (36000d3100314f600000000000000018a) dm-2 COMPELNT,Compellent Vol
size=60G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active
1:0:0:1 sdb 8:16 active undef running
1:0:3:1 sdd 8:48 active undef running
8:0:0:1 sdc 8:32 active undef running
8:0:3:1 sde 8:64 active undef running

If I look at the devices that the multipathd output shows them as (sdb-sdd) [root@div18oscomp2 by-path]# ls -l | grep sd[b-e]
lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.0-fc-0x5000d3100314f624-lun-1 -> ../../sdd lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.0-fc-0x5000d3100314f626-lun-1 -> ../../sdb lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.1-fc-0x5000d3100314f623-lun-1 -> ../../sde lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.1-fc-0x5000d3100314f625-lun-1 -> ../../sdc

Yet, in the below log, its looking for a different device path Looking for Fibre Channel dev /dev/disk/by-path/pci-0000:04:00.1-fc-0x5000d3100314f624-lun-5

Why is nova looking for lun5, when the volume is on lun-1???? Where does it get lun5 from??

Any help would be appreciated

nova-compute log shows: 2017-04-10 15:08:51.325 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 7a593c6f571645a79159d376df77d425 exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-04-10 15:08:51.343 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 7a593c6f571645a79159d376df77d425 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296 2017-04-10 15:08:51.800 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: 8ad49bc12bfd4316a1bfe83e00de9c98 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:196 2017-04-10 15:08:51.812 18896 DEBUG oslo_messaging._drivers.amqpdriver [req-75d5269a-de16-4586-8c13-477ea3d446a4 c42f3db35ab54eb69f2c829523c81668 b713ca31297341ad934ad5a19f2ab467 - - -] CALL msg_id: ccea24299fac4ba98ac4c6d28654ed2e exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-04-10 15:08:51.827 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: ccea24299fac4ba98ac4c6d28654ed2e __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296 2017-04-10 15:08:51.829 18896 DEBUG nova.objects.instance [req-75d5269a-de16-4586-8c13-477ea3d446a4 c42f3db35ab54eb69f2c829523c81668 b713ca31297341ad934ad5a19f2ab467 - - -] Lazy-loading 'flavor' on Instance uuid f248c4bb-95f7-489e-9442-2d5f7588c2c9 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2017-04-10 15:08:51.831 18896 DEBUG oslo_messaging._drivers.amqpdriver [req-75d5269a-de16-4586-8c13-477ea3d446a4 c42f3db35ab54eb69f2c829523c81668 b713ca31297341ad934ad5a19f2ab467 - - -] CALL msg_id: 2206f6bd1d054150af801dc9e62a978c exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-04-10 15:08:51.884 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2206f6bd1d054150af801dc9e62a978c __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296 2017-04-10 ... (more)

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2017-04-11 08:01:56 -0500

lyarwood gravatar image

It appears that the sd[b-e] path devices are stale, can you flush the 36000d3100314f600000000000000018a device or restart the compute node?

Why is nova looking for lun5, when the volume is on lun-1???? Where does it get lun5 from??

The connection_info returned from cinder-volume for the volume lists lun-5 :

'connection_info': {u'driver_volume_type': u'fibre_channel', [..], u'serial': u'ede15f11-998d-4c58-9872-a5ca78507235', u'data': {u'initiator_target_map': {u'2100001B3287FA25': [u'5000D3100314F626', u'5000D3100314F624'], u'2101001B32A7FA25': [u'5000D3100314F623', u'5000D3100314F625']} [..], u'target_lun': 5,[..]}, [..]} _get_guest_xml
edit flag offensive delete link more

Comments

I tried multpath -F, and also rebooted the host. The san still mapped the volume as lun 1 over the fibre channel, and nova continued to try to find it at lun5. I actually had to unmap it from the Dell San Side, and remap, specifying to use lun5. Then nova would find the volume and start the vm.

jkilborn gravatar imagejkilborn ( 2017-04-13 08:16:50 -0500 )edit

Still don't know where nova gets the connection_info from ... Its not stored in cinder db or nova db.

jkilborn gravatar imagejkilborn ( 2017-04-13 08:17:35 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2017-04-10 15:20:12 -0500

Seen: 266 times

Last updated: Apr 11 '17