Manila with CephFS: Error in mkdir when creating shares

asked 2020-01-09 09:54:28 -0500

Fenuks gravatar image

Hello. I need help with the following problem.
Installed Manila with CephFS NFS shares as per and run into error

Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")

when creating new share both via manila cli and via Horizon.

It looks like 'permission denied' error, but client.manila in ceph should have all the required permissions (see below).
Mounting the filesystem manually and creating files and folders inside works fine:

ceph-fuse /mnt/cephfs --id=manila --cluster=ceph --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/ceph.client.manila.keyring --client-mountpoint=/
$ mount | grep '/mnt/cephfs'
ceph-fuse on /mnt/cephfs type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
$ mkdir -p /mnt/cephfs/volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b; echo $?

Not sure where to look further and what I might've done wrong. Need help!

My setup:

Openstack 19.0.8 installed with openstack-ansible. Ceph MDS installed via ceph-ansible. Manila CephFS backend configured mostly manually, as openstack-ansible does not seem to have such an option fully available.


cephfs_auth_id = manila
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_enable_snapshots = False
cephfs_ganesha_server_ip =
cephfs_protocol_helper_type = NFS
driver_handles_share_servers = False
filter_function = share.size >= 0
goodness_function = share.size >= 0
share_backend_name = CEPHFSNFS1
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver

Files and permissions:

$ ls -l /etc/ceph/
-rw-r----- 1 root cephkeys 82 Dec 23 05:57 /etc/ceph/ceph.client.manila.keyring
-rw-r--r-- 1 root root 796 Jan  6 15:34 /etc/ceph/ceph.conf
$ getent group cephkeys

Auth settings:

$ ceph auth get client.manila
    key = [redacted]
    caps mds = "allow *"
    caps mon = "allow r, allow command \"auth del\", allow command \"auth caps\", allow command \"auth get\", allow command \"auth get-or-create\""
    caps osd = "allow rw"


$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
$ ceph mds stat
cephfs-1/1/1 up  {0=ceph1=up:active}, 1 up:standby

Error log:

DEBUG manila.share.drivers.cephfs.driver [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] create_share CEPHFSNFS1 name=1d20ed2f-a8f9-48ea-94e1-94722829b29b size=1 share_group_id=None create_share /openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/manila/share/drivers/cephfs/
INFO ceph_volume_client [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] create_volume: /volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b
ERROR manila.share.manager [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Share instance 1d20ed2f-a8f9-48ea-94e1-94722829b29b failed on creation.: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
WARNING manila.share.manager [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
INFO manila.message.api [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Creating message record for request_id = req-4306125c-204c-442d-a1bf-e7fdd58bb77e
ERROR oslo_messaging.rpc.server [req-4306125c-204c-442d-a1bf-e7fdd58bb77e e7873600a6065c26a0a49eec50b400ec0d8be90aa58fe3e52de75847da61aa95 7ddc72f94aef4ad6956a47c66511f611 - - -] Exception during message handling: Error: (13, "error in mkdir 'volumes/_nogroup/1d20ed2f-a8f9-48ea-94e1-94722829b29b': error code 13")
    ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    ERROR oslo_messaging.rpc.server   File "/openstack/venvs/manila-19.0.8/lib/python2.7/site-packages/oslo_messaging/rpc/", line 166, in _process_incoming
    ERROR oslo_messaging.rpc.server     res = self ...
edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted

answered 2020-02-15 03:36:35 -0500

Fenuks gravatar image

It turned out to be POSIX permission error in the mounted CEPHFS.

When manila-share creates shares, it mounts cephfs into temporary directory starting at it's / and then proceeds to create regular directories inside. But the problem is that manila-share is running as user:group manila:manila while cephfs / belongs to root:root with permissions 755.

Mounting cephfs manually and doing chown manila:manila before creating shares resolved this error.

Setting those in /etc/ceph/ceph.conf:

client mount uid = 0
client mount gid = 0

does not do anything for my case, mounted cephfs directories still belong to root. Might be a bug.

P.S. After this error I've stumbled into a new ones, including possible errors in implementation of cephfs driver in manila-share. But this one is closed.

edit flag offensive delete link more


whether driver initilalisation error was fixed ?if yes can you share how ..We are also facing same issue

Deepa gravatar imageDeepa ( 2020-05-25 12:47:15 -0500 )edit

There was no driver error, I was wrong about it. Metadata in case of nogroup is indeed saved as _:<id>.meta, not _nogroup/<id>.meta.

Fenuks gravatar imageFenuks ( 2020-05-27 01:02:45 -0500 )edit

Regarding permissions on manila share, we still run installation where we changed owner manually and without client mount uid/gid lines in ceph.conf and it is working — creating, deleting and using new shares all is fine. Adding them now might break our installation.

That is the only fix we have.

Fenuks gravatar imageFenuks ( 2020-05-27 01:07:53 -0500 )edit

thanks much for the reply ..Seems we are receiving a different error driver initialisation error Error encountered during initialization of driver CephFSDriver@xxxxx: IndexError: list index out of range

Deepa gravatar imageDeepa ( 2020-05-27 08:53:03 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2020-01-09 09:54:28 -0500

Seen: 180 times

Last updated: Feb 15