Ask Your Question

Help! nova can't write to /var/lib/nova/instances if it's on a separate LVM mount point

asked 2018-11-01 09:26:19 -0600

brainbarker gravatar image

updated 2018-11-02 09:25:01 -0600

To make more ephemeral storage available on my compute nodes, I mounted a large LVM volume over the /var/lib/nova/instances directory. The mount seems to work fine, and the nova user can read/write/delete files on the drive (owner and group are nova for the entire drive), but the compute service can not write files there. The error message I get complains about deleting files, but there are never any files written there during instance creation at all.

It must be some obscure permissions issue, but I can't track it down. Any suggestions would be greatly appreciated.

Many thanks,

Brian Baker

Testing that the nova user can access the mounted drive:

[root@bcompute2 ~]# sudo -u nova bash
bash-4.2$ ls -l /var/lib/nova/instances
total 28
drwxr-xr-x. 2 nova nova  4096 Oct 30 12:32 _base
-rw-r--r--. 1 nova nova    32 Oct 31 14:27 compute_nodes
drwxr-xr-x. 2 nova nova  4096 Oct 30 12:31 locks
drwx------. 2 nova nova 16384 Oct 31 13:41 lost+found
bash-4.2$ cd /var/lib/nova/instances
bash-4.2$ mkdir foo
bash-4.2$ touch foo/bar
bash-4.2$ rm foo/bar
bash-4.2$ rmdir foo
bash-4.2$ exit

The error in nova-compute.log is:

"Instance failed to spawn: libvirtError: Unable to delete file /var/lib/nova/instances/24e02d3b-941c-4b11-b5dd-3bd5ef5dae94/console.log: Permission denied"

The mount command shows:

/dev/mapper/ephemeralvg-ephemeral on /var/lib/nova/instances type ext4 (rw,relatime,seclabel,data=ordered)

The entry in fstab is:

/dev/ephemeralvg/ephemeral /var/lib/nova/instances ext4 defaults 0 0

The mounted volume looks like:

  --- Logical volume ---
  LV Path                /dev/ephemeralvg/ephemeral
  LV Name                ephemeral
  VG Name                ephemeralvg
  LV UUID                wM2r5c-WIBK-KPxL-6bE3-YmEV-VFun-L0o04l
  LV Write Access        read/write
  LV Creation host, time bcompute2, 2018-10-31 13:40:00 -0400
  LV Status              available
  # open                 0
  LV Size                <372.62 GiB
  Current LE             95390
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted

answered 2018-11-02 09:27:02 -0600

brainbarker gravatar image

OK, so I'm answering my own question. :-)

After digging through /var/log/messages and /var/log/audit/audit.log, and doing a bit more googling, I figured out that I had left SELinux in "enforcing" mode. It needs to be in "permissive" mode to allow the mounted drive to be used by virtlogd. [Maybe there is a way to fix this without going to permissive mode, but that was the easiest fix in my situation].

I'm including the instructions I used to set this up, including this fix, in case anyone is interested.



# OPTIONAL: Utilize extra drives on compute node for ephemeral storage

# You will need to have SELinux in permissive mode to access the new disk:

# Edit /etc/selinux/config and set SELINUX=permissive

# Then set for autorelabel of filesystems and reboot:
touch /.autorelabel

# Now you can set up the new disk:

# Find availables drives

# Create an LVM volume containing all drives to be used for ephemeral storage
# (If pvcreate fails, make sure the drives are not being used, and then use fdisk to
# delete the partitions)
pvcreate -v /dev/{sda,sdb}
vgcreate -v ephemeralvg /dev/{sda,sdb}
lvcreate -l 100%FREE -n ephemeral ephemeralvg
mkfs.xfs -L ephemeral /dev/ephemeralvg/ephemeral

# Stop the compute services
systemctl stop libvirtd.service openstack-nova-compute.service

# If there are existing instances, move them to the new disk
mkdir /mnt/_eph
mount /dev/ephemeralvg/ephemeral /mnt/_eph
chown -R nova:nova /mnt/_eph
rsync -a /var/lib/nova/instances/ /mnt/_eph/
umount /dev/ephemeralvg/ephemeral
rmdir /mnt/_eph

# If you're feeling bold, reclaim the previous ephemeral storage
rm -rf /var/lib/nova/instances/*

# Add to /etc/fstab
/dev/ephemeralvg/ephemeral /var/lib/nova/instances xfs defaults 0 0

# Mount volume and and verify
mount -a
ls /var/lib/nova/instances
df -k /var/lib/nova/instances

# Restart the compute services
systemctl restart libvirtd.service openstack-nova-compute.service
edit flag offensive delete link more

answered 2018-11-02 05:21:55 -0600

yasin lachiny gravatar image

updated 2018-11-02 05:24:48 -0600 by this url you can create lvm and install cinder-volume and config /etc/cinder/cinder.conf after that stop all cinder volume exept this one and build your instance.

edit flag offensive delete link more


Good link, but that refers to using LVM as a Cinder backend. I needed an LVM volume mounted as the compute node's ephemeral storage.

brainbarker gravatar imagebrainbarker ( 2018-11-02 09:16:35 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2018-11-01 09:26:19 -0600

Seen: 991 times

Last updated: Nov 02 '18