Help! nova can't write to /var/lib/nova/instances if it's on a separate LVM mount point
To make more ephemeral storage available on my compute nodes, I mounted a large LVM volume over the /var/lib/nova/instances directory. The mount seems to work fine, and the nova user can read/write/delete files on the drive (owner and group are nova for the entire drive), but the compute service can not write files there. The error message I get complains about deleting files, but there are never any files written there during instance creation at all.
It must be some obscure permissions issue, but I can't track it down. Any suggestions would be greatly appreciated.
Many thanks,
Brian Baker
Testing that the nova user can access the mounted drive:
[root@bcompute2 ~]# sudo -u nova bash
bash-4.2$ ls -l /var/lib/nova/instances
total 28
drwxr-xr-x. 2 nova nova 4096 Oct 30 12:32 _base
-rw-r--r--. 1 nova nova 32 Oct 31 14:27 compute_nodes
drwxr-xr-x. 2 nova nova 4096 Oct 30 12:31 locks
drwx------. 2 nova nova 16384 Oct 31 13:41 lost+found
bash-4.2$ cd /var/lib/nova/instances
bash-4.2$ mkdir foo
bash-4.2$ touch foo/bar
bash-4.2$ rm foo/bar
bash-4.2$ rmdir foo
bash-4.2$ exit
The error in nova-compute.log is:
"Instance failed to spawn: libvirtError: Unable to delete file /var/lib/nova/instances/24e02d3b-941c-4b11-b5dd-3bd5ef5dae94/console.log: Permission denied"
The mount command shows:
/dev/mapper/ephemeralvg-ephemeral on /var/lib/nova/instances type ext4 (rw,relatime,seclabel,data=ordered)
The entry in fstab is:
/dev/ephemeralvg/ephemeral /var/lib/nova/instances ext4 defaults 0 0
The mounted volume looks like:
--- Logical volume ---
LV Path /dev/ephemeralvg/ephemeral
LV Name ephemeral
VG Name ephemeralvg
LV UUID wM2r5c-WIBK-KPxL-6bE3-YmEV-VFun-L0o04l
LV Write Access read/write
LV Creation host, time bcompute2, 2018-10-31 13:40:00 -0400
LV Status available
# open 0
LV Size <372.62 GiB
Current LE 95390
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2
