"nova start" fails to find cinder volume, yet volume is attached to node
Upgraded from liberty to newton. Trying to start vm back up on upgraded compute node. The vm was shutdown, then the update was performed, now I can't start that vm. I can create a new vm, cinder backed, and it creates the volume on the dell san , and attaches to host, and starts fine. However, the existing VM keeps failing to start.
The device in the libvirt.xml shows the disk dev as :
source dev="/dev/disk/by-id/dm-uuid-mpath-36000d3100314f600000000000000018a"/
and multipath -l shows the device as well
mpathcd (36000d3100314f600000000000000018a) dm-2 COMPELNT,Compellent Vol
size=60G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
1:0:0:1 sdb 8:16 active undef running
1:0:3:1 sdd 8:48 active undef running
8:0:0:1 sdc 8:32 active undef running
8:0:3:1 sde 8:64 active undef running
If I look at the devices that the multipathd output shows them as (sdb-sdd)
[root@div18oscomp2 by-path]# ls -l | grep sd[b-e]
lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.0-fc-0x5000d3100314f624-lun-1 -> ../../sdd
lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.0-fc-0x5000d3100314f626-lun-1 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.1-fc-0x5000d3100314f623-lun-1 -> ../../sde
lrwxrwxrwx. 1 root root 9 Apr 10 06:13 pci-0000:04:00.1-fc-0x5000d3100314f625-lun-1 -> ../../sdc
Yet, in the below log, its looking for a different device path Looking for Fibre Channel dev /dev/disk/by-path/pci-0000:04:00.1-fc-0x5000d3100314f624-lun-5
Why is nova looking for lun5, when the volume is on lun-1???? Where does it get lun5 from??
Any help would be appreciated
nova-compute log shows: 2017-04-10 15:08:51.325 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 7a593c6f571645a79159d376df77d425 exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-04-10 15:08:51.343 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 7a593c6f571645a79159d376df77d425 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296 2017-04-10 15:08:51.800 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: 8ad49bc12bfd4316a1bfe83e00de9c98 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:196 2017-04-10 15:08:51.812 18896 DEBUG oslo_messaging._drivers.amqpdriver [req-75d5269a-de16-4586-8c13-477ea3d446a4 c42f3db35ab54eb69f2c829523c81668 b713ca31297341ad934ad5a19f2ab467 - - -] CALL msg_id: ccea24299fac4ba98ac4c6d28654ed2e exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-04-10 15:08:51.827 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: ccea24299fac4ba98ac4c6d28654ed2e __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296 2017-04-10 15:08:51.829 18896 DEBUG nova.objects.instance [req-75d5269a-de16-4586-8c13-477ea3d446a4 c42f3db35ab54eb69f2c829523c81668 b713ca31297341ad934ad5a19f2ab467 - - -] Lazy-loading 'flavor' on Instance uuid f248c4bb-95f7-489e-9442-2d5f7588c2c9 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2017-04-10 15:08:51.831 18896 DEBUG oslo_messaging._drivers.amqpdriver [req-75d5269a-de16-4586-8c13-477ea3d446a4 c42f3db35ab54eb69f2c829523c81668 b713ca31297341ad934ad5a19f2ab467 - - -] CALL msg_id: 2206f6bd1d054150af801dc9e62a978c exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-04-10 15:08:51.884 18896 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2206f6bd1d054150af801dc9e62a978c __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296 2017-04-10 ...