Cinder: Lost /dev/disks/by-path links

asked 2016-12-17 06:27:55 -0600

KeepTriin gravatar image

updated 2016-12-20 06:52:53 -0600

Blockquote

During a recent update of servers, (including firmware and BIOS updates), selinux was changed to enforcing and OpenStack was restarted. All processes were found to be active, so the servers were rebooted. Following the reboot, all OpenStack processes were one again found to be active (and stable). However, when attempting to start an instance using block storage (LVM/ISCSI using lioadm) this failed to find the block storage path under /dev/disks/by-path. further investigation showed that PCI devices were listed under the path, but not ISCSI targets. It is possible to list ISCSI objects using :

targetcli ls and iscsiadm -m node ...

and its also possible to see the block storage targets using the ls, LVM and mapper commands

ls -l /dev/mapper
ls -l /dev/vg_cinder/*
lvs
dmsetup ls

Etc...

So I should be able to recreate the entries under /dev/disks manually. However these aren't persistent across reboots and are not being recreated by:

udevadm trigger

I suspect that the cinder/nova processes are somehow responsible for the management of the links and that in implementing selinux in enforcing mode I've managed to drop those links and have those changes committed to a DB table so they're no longer associated with an instance. Could someone explain how these links are managed by OpenStack and how I might recreate these links to restore links that are persistently managed over reboots?

Update: This appears to be an issue with the filtering of guest volume groups from the host machine when the guest VG is created on an additional volume.

e.g 
create a volume in openStack
Attach to instance,  -> presented as /dev/vdb, 
On the instance: 
pvcreate /dev/vdb
vgcreate vg_name /dev/vdb

Then, if/when the LV metadata gets rescanned on the host, it will find the VG UUID, but not the PV UUID, so you get behaviour like that discussed here:

https://forum.proxmox.com/threads/lvm...

and symptomatic errors of:

Couldn't find device with uuid '7RzjNo-PSq0-NjEc-EFGf-luil-jmA3-c6nGrW'

My block storage devices are all presented to the host and so any scan will find them. This error however, prevents iscsi from accessing the block device on the remote node and so the /dev/disk/by-path links are not created. At present I haven't found a method of excluding these embedded volumes from the scan via lvm.conf filtering. If I do, I'll post this back here in case it's of any use to anyone else. If anyone else knows how to do this, I'd appreciate their input. My current filter looks like this to include the internal disks on the host and to exclude anything else. However, on this small system the additional cinder storage is created as VG on those disks.

filter = [ "a|^/dev/sda|", "a|^/dev/sdb|", "r|.*|"]

Thanks

edit retag flag offensive close merge delete