Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Help! nova can't write to /var/lib/nova/instances if it's on a separate LVM mount point

To make more ephemeral storage available on my compute nodes, I mounted a large LVM volume over the /var/lib/nova/instances directory. The mount seems to work fine, and the nova user can read/write/delete files on the drive (owner and group are nova for the entire drive), but the compute service can not write files there. The error message I get complains about deleting files, but there are never any files written there during instance creation at all.

It must be some obscure permissions issue, but I can't track it down. Any suggestions would be greatly appreciated.

Many thanks,

Brian Baker


Testing that the nova user can access the mounted drive:

[root@bcompute2 ~]# sudo -u nova bash
bash-4.2$ ls -l /var/lib/nova/instances
total 28
drwxr-xr-x. 2 nova nova  4096 Oct 30 12:32 _base
-rw-r--r--. 1 nova nova    32 Oct 31 14:27 compute_nodes
drwxr-xr-x. 2 nova nova  4096 Oct 30 12:31 locks
drwx------. 2 nova nova 16384 Oct 31 13:41 lost+found
bash-4.2$ cd /var/lib/nova/instances
bash-4.2$ mkdir foo
bash-4.2$ touch foo/bar
bash-4.2$ rm foo/bar
bash-4.2$ rmdir foo
bash-4.2$ exit

The error in nova-compute.log is:

"Instance failed to spawn: libvirtError: Unable to delete file /var/lib/nova/instances/24e02d3b-941c-4b11-b5dd-3bd5ef5dae94/console.log: Permission denied"

The mount command shows:

/dev/mapper/ephemeralvg-ephemeral on /var/lib/nova/instances type ext4 (rw,relatime,seclabel,data=ordered)

The entry in fstab is:

/dev/ephemeralvg/ephemeral /var/lib/nova/instances ext4 defaults 0 0

The mounted volume looks like:

  --- Logical volume ---
  LV Path                /dev/ephemeralvg/ephemeral
  LV Name                ephemeral
  VG Name                ephemeralvg
  LV UUID                wM2r5c-WIBK-KPxL-6bE3-YmEV-VFun-L0o04l
  LV Write Access        read/write
  LV Creation host, time bcompute2, 2018-10-31 13:40:00 -0400
  LV Status              available
  # open                 0
  LV Size                <372.62 GiB
  Current LE             95390
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

Help! nova can't write to /var/lib/nova/instances if it's on a separate LVM mount point

To make more ephemeral storage available on my compute nodes, I mounted a large LVM volume over the /var/lib/nova/instances directory. The mount seems to work fine, and the nova user can read/write/delete files on the drive (owner and group are nova for the entire drive), but the compute service can not write files there. The error message I get complains about deleting files, but there are never any files written there during instance creation at all.

It must be some obscure permissions issue, but I can't track it down. Any suggestions would be greatly appreciated.

Many thanks,

Brian Baker


Testing that the nova user can access the mounted drive:

[root@bcompute2 ~]# sudo -u nova bash
bash-4.2$ ls -l /var/lib/nova/instances
total 28
drwxr-xr-x. 2 nova nova  4096 Oct 30 12:32 _base
-rw-r--r--. 1 nova nova    32 Oct 31 14:27 compute_nodes
drwxr-xr-x. 2 nova nova  4096 Oct 30 12:31 locks
drwx------. 2 nova nova 16384 Oct 31 13:41 lost+found
bash-4.2$ cd /var/lib/nova/instances
bash-4.2$ mkdir foo
bash-4.2$ touch foo/bar
bash-4.2$ rm foo/bar
bash-4.2$ rmdir foo
bash-4.2$ exit

The error in nova-compute.log is:

"Instance failed to spawn: libvirtError: Unable to delete file /var/lib/nova/instances/24e02d3b-941c-4b11-b5dd-3bd5ef5dae94/console.log: Permission denied"

The mount command shows:

/dev/mapper/ephemeralvg-ephemeral on /var/lib/nova/instances type ext4 (rw,relatime,seclabel,data=ordered)

The entry in fstab is:

/dev/ephemeralvg/ephemeral /var/lib/nova/instances ext4 defaults 0 0

The mounted volume looks like:

  --- Logical volume ---
  LV Path                /dev/ephemeralvg/ephemeral
  LV Name                ephemeral
  VG Name                ephemeralvg
  LV UUID                wM2r5c-WIBK-KPxL-6bE3-YmEV-VFun-L0o04l
  LV Write Access        read/write
  LV Creation host, time bcompute2, 2018-10-31 13:40:00 -0400
  LV Status              available
  # open                 0
  LV Size                <372.62 GiB
  Current LE             95390
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2