Manila with CephFS: Error in mkdir when creating shares

asked 2020-01-09 09:54:28 -0600

Fenuks gravatar image

Hello. I need help with the following problem.
Installed Manila with CephFS NFS shares as per https://docs.openstack.org/manila/ste... and run into error

Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")

when creating new share both via manila cli and via Horizon.

It looks like 'permission denied' error, but client.manila in ceph should have all the required permissions (see below).
Mounting the filesystem manually and creating files and folders inside works fine:

ceph-fuse /mnt/cephfs --id=manila --cluster=ceph --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/ceph.client.manila.keyring --client-mountpoint=/
…
$ mount | grep '/mnt/cephfs'
ceph-fuse on /mnt/cephfs type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
$ mkdir -p /mnt/cephfs/volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b; echo $?
0

Not sure where to look further and what I might've done wrong. Need help!

My setup:

Openstack 19.0.8 installed with openstack-ansible. Ceph MDS installed via ceph-ansible. Manila CephFS backend configured mostly manually, as openstack-ansible does not seem to have such an option fully available.

manila.conf:

[cephfsnfs1]
cephfs_auth_id = manila
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_enable_snapshots = False
cephfs_ganesha_server_ip = 192.168.0.240
cephfs_protocol_helper_type = NFS
driver_handles_share_servers = False
filter_function = share.size >= 0
goodness_function = share.size >= 0
share_backend_name = CEPHFSNFS1
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver

Files and permissions:

$ ls -l /etc/ceph/
-rw-r----- 1 root cephkeys 82 Dec 23 05:57 /etc/ceph/ceph.client.manila.keyring
-rw-r--r-- 1 root root 796 Jan  6 15:34 /etc/ceph/ceph.conf
$ getent group cephkeys
cephkeys:x:1001:cinder,nova,manila

Auth settings:

$ ceph auth get client.manila
[client.manila]
    key = [redacted]
    caps mds = "allow *"
    caps mon = "allow r, allow command \"auth del\", allow command \"auth caps\", allow command \"auth get\", allow command \"auth get-or-create\""
    caps osd = "allow rw"

CephFS:

$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
$ ceph mds stat
cephfs-1/1/1 up  {0=ceph1=up:active}, 1 up:standby

Error log:

DEBUG manila.share.drivers.cephfs.driver [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] create_share CEPHFSNFS1 name=1d20ed2f-a8f9-48ea-94e1-94722829b29b size=1 share_group_id=None create_share /openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py:246
INFO ceph_volume_client [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] create_volume: /volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b
ERROR manila.share.manager [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Share instance 1d20ed2f-a8f9-48ea-94e1-94722829b29b failed on creation.: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
WARNING manila.share.manager [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
INFO manila.message.api [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Creating message record for request_id = req-4306125c-204c-442d-a1bf-e7fdd58bb77e
ERROR oslo_messaging.rpc.server [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Exception during message handling: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
    ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
    ERROR oslo_messaging.rpc.server     res = self ...
(more)
edit retag flag offensive close merge delete