Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Manila with CephFS: Error in mkdir when creating shares

Hello. I need help with the following problem.
Installed Manila with CephFS NFS shares as per https://docs.openstack.org/manila/stein/admin/cephfs_driver.html and run into error

Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")

when creating new share both via manila cli and via Horizon.

It looks like 'permission denied' error, but client.manila in ceph should have all the required permissions (see below).
Mounting the filesystem manually and creating files and folders inside works fine:

ceph-fuse /mnt/cephfs --id=manila --cluster=ceph --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/ceph.client.manila.keyring --client-mountpoint=/
…
$ mount | grep '/mnt/cephfs'
ceph-fuse on /mnt/cephfs type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
$ mkdir -p /mnt/cephfs/volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b; echo $?
0

Not sure where to look further and what I might've done wrong. Need help!

My setup:

Openstack 19.0.8 installed with openstack-ansible. Ceph MDS installed via ceph-ansible. Manila CephFS backend configured mostly manually, as openstack-ansible does not seem to have such an option fully available.

manila.conf:

[cephfsnfs1]
cephfs_auth_id = manila
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_enable_snapshots = False
cephfs_ganesha_server_ip = 192.168.0.240
cephfs_protocol_helper_type = NFS
driver_handles_share_servers = False
filter_function = share.size >= 0
goodness_function = share.size >= 0
share_backend_name = CEPHFSNFS1
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver

Files and permissions:

$ ls -l /etc/ceph/
-rw-r----- 1 root cephkeys 82 Dec 23 05:57 /etc/ceph/ceph.client.manila.keyring
-rw-r--r-- 1 root root 796 Jan  6 15:34 /etc/ceph/ceph.conf
$ getent group cephkeys
cephkeys:x:1001:cinder,nova,manila

Auth settings:

$ ceph auth get client.manila
[client.manila]
    key = [redacted]
    caps mds = "allow *"
    caps mon = "allow r, allow command \"auth del\", allow command \"auth caps\", allow command \"auth get\", allow command \"auth get-or-create\""
    caps osd = "allow rw"

CephFS:

$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
$ ceph mds stat
cephfs-1/1/1 up  {0=ceph1=up:active}, 1 up:standby

Error log:

DEBUG manila.share.drivers.cephfs.driver [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] create_share CEPHFSNFS1 name=1d20ed2f-a8f9-48ea-94e1-94722829b29b size=1 share_group_id=None create_share /openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py:246
INFO ceph_volume_client [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] create_volume: /volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b
ERROR manila.share.manager [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Share instance 1d20ed2f-a8f9-48ea-94e1-94722829b29b failed on creation.: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
WARNING manila.share.manager [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
INFO manila.message.api [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Creating message record for request_id = req-4306125c-204c-442d-a1bf-e7fdd58bb77e
ERROR oslo_messaging.rpc.server [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Exception during message handling: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
    ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
    ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
    ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
    ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/manager.py", line 186, in wrapped
    ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/utils.py", line 568, in wrapper
    ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/manager.py", line 1772, in create_share_instance
    ERROR oslo_messaging.rpc.server     exception=e)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    ERROR oslo_messaging.rpc.server     self.force_reraise()
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/manager.py", line 1735, in create_share_instance
    ERROR oslo_messaging.rpc.server     context, share_instance, share_server=share_server)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py", line 256, in create_share
    ERROR oslo_messaging.rpc.server     mode=self._cephfs_volume_mode)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/ceph_volume_client.py", line 628, in create_volume
    ERROR oslo_messaging.rpc.server     self._mkdir_p(path, mode)
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/ceph_volume_client.py", line 609, in _mkdir_p
    ERROR oslo_messaging.rpc.server     self.fs.mkdir(subpath, mode)
    ERROR oslo_messaging.rpc.server   File "cephfs.pyx", line 724, in cephfs.LibCephFS.mkdir
    ERROR oslo_messaging.rpc.server Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
    ERROR oslo_messaging.rpc.server