Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

**libvirtError: Cannot access storage file XXX (as uid:107, gid:107): No such file or directory**

Sorry for crossposting, Oracle community seems the wrong place to ask...

I've installed an OpenStack POC with Oracle OpenStack (using kolla).

I tested cinder volume attachment with the default lvm-driver and if works smoothly.

I need to use nfs4 (on Netapp and probably on other NAS providers), so I configured the cinder nfs driver :

[root@oos-master config]# cat /etc/kolla/config/cinder.conf

[DEFAULT]
enabled_backends = lvmdriver-1,nfs-1


[nfs-1]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
#nfs_mount_options = "vers=3,nolock"
nfs_shares_config = /etc/cinder/nfs_shares
nfs_sparsed_volumes = True
volume_backend_name=nfs-1

and redeployed the cinder service and create by hand in the docker container the file /etc/cinder/nfs_shares:

[root@oos-ctrl1 ~]# docker exec -it cinder_volume bash
[root@oos-ctrl1 /]# cat /etc/cinder/nfs_shares
<OMISSIS>:/volOOSTest
[root@oos-ctrl1 /]# exit

I do manage also to create a volume on nfs, BUT... when I try to attach this volume on a instance it does not succed and I get an error in the nova log service docker container:

# here the vm:
#
[root@oos-master config]# docker-ostk nova list
+--------------------------------------+--------------------+---------+------------+-------------+----------------------+
| ID                                   | Name               | Status  | Task State | Power State | Networks             |
+--------------------------------------+--------------------+---------+------------+-------------+----------------------+
| 03d5906f-18ee-40ce-98a4-03e4fe1473c3 | cirros-instance--5 | ACTIVE  | -          | Running     | test=192.168.235.101 |
+--------------------------------------+--------------------+---------+------------+-------------+----------------------+

# here the volumes, the first one is on nfs:
#
[root@oos-master config]# docker-ostk nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| ID                                   | Status    | Display Name | Size | Volume Type | Attached to                          |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| 6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa | available |              | 5    | nfs         |                                      |
| f24a3fe0-d994-49f6-be4c-6e62c021fea6 | available |              | 1    | lvm         |                                      |
| e539e48a-788c-4e75-b081-4641648f909b | available |              | 1    | -           | 6f630fc1-0299-4b0f-b8b9-77bb98f3cf94 |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+

trying to attach the nfs volume on a instance

#
[root@oos-master config]# docker-ostk nova volume-attach 03d5906f-18ee-40ce-98a4-03e4fe1473c3 6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa |
| serverId | 03d5906f-18ee-40ce-98a4-03e4fe1473c3 |
| volumeId | 6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa |
+----------+--------------------------------------+

... but the volume is not attached

#
[root@oos-master config]# docker-ostk nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| ID                                   | Status    | Display Name | Size | Volume Type | Attached to                          |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| 6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa | available |              | 5    | nfs         |                                      |
| f24a3fe0-d994-49f6-be4c-6e62c021fea6 | available |              | 1    | lvm         |                                      |
| e539e48a-788c-4e75-b081-4641648f909b | available |              | 1    | -           | 6f630fc1-0299-4b0f-b8b9-77bb98f3cf94 |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
[root@oos-master config]#

and in the nova service log I get:

2016-03-29 09:47:29.335 1 INFO nova.compute.manager [req-88e88adb-eeaa-4e94-a21e-26daff128281 fb769977a9454ec3a1d27ad2d3757a2e 7fa49cadeda84a27b9b9c38e813ac826 - - -] [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3] Attaching volume 6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa to /dev/vdb
2016-03-29 09:47:30.910 1 ERROR nova.virt.libvirt.driver [req-88e88adb-eeaa-4e94-a21e-26daff128281 fb769977a9454ec3a1d27ad2d3757a2e 7fa49cadeda84a27b9b9c38e813ac826 - - -] [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3] Failed to attach volume at mountpoint: /dev/vdb
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3] Traceback (most recent call last):
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1092, in attach_volume
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]     virt_dom.attachDeviceFlags(conf.to_xml(), flags)
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]     rv = execute(f, *args, **kwargs)
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]     six.reraise(c, e, tb)
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]     rv = meth(*args, **kwargs)
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 554, in attachDeviceFlags
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-29 09:47:30.910 1 TRACE nova.virt.libvirt.driver [instance: 03d5906f-18ee-40ce-98a4-03e4fe1473c3] **libvirtError: Cannot access storage file '/var/lib/nova/mnt/6c90c142f085ee7d319fb9f780900a21/volume-6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa' (as uid:107, gid:107): No such file or directory**

The nfs share is correctly configured. To test it, on the nova container, I changed the shell of qemu to /bin/bash, mount the nfs share, and test the accessibility of the cinder volume:

[root@oos-compute2 ~]# docker exec -it nova_compute bash
[root@oos-compute2 /]# grep qemu /etc/passwd
qemu:x:107:107:qemu user:/:/bin/bash
[root@oos-compute2 /]# mount <OMISSIS>:/volOOSTest /mnt/a   
[root@oos-compute2 /]# su - qemu                             
Last login: Tue Mar 29 09:55:40 UTC 2016 on pts/3
-bash-4.2$ cd /mnt/a
-bash-4.2$ stat volume-6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa
  File: 'volume-6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa'
  Size: 5368709120 Blocks: 0          IO Block: 65536  regular file
Device: cdh/205d Inode: 99          Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-03-29 10:09:25.922660000 +0000
Modify: 2016-03-28 21:46:15.873116000 +0000
Change: 2016-03-28 21:46:16.126188000 +0000
Birth: -
-bash-4.2$ dd if=volume-6a7fd5d3-02d9-47a9-be2f-1bff71c28aaa of=/dev/null
10485760+0 records in
10485760+0 records out
5368709120 bytes (5.4 GB) copied, 22.2864 s, 241 MB/s