unable to modify user capabilities in ceph
I have trouble connecting cinder to ceph. Error messages from the cinder logs:
Update driver status failed: (config name rbd-1) is uninitialized.
Error connecting to ceph cluster.: rados.ObjectNotFound: [errno 2] error connecting to the cluster
Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster
I believe this is due to wrong permissions on the ceph side.
Current permissions:
client.cinder
caps: [mon] profile rbd
caps: [osd] profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images
client.cinder-backup
caps: [mon] profile rbd
caps: [osd] profile rbd pool=backups
According to the https://docs.ceph.com/docs/nautilus/rbd/rbd-openstack/#setup-ceph-client-authentication (ceph documentation), which is https://docs.openstack.org/kolla-ansible/train/reference/storage/external-ceph-guide.html (referenced by the OpenStack documentation), the permissions should be created like this:
ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images' mgr 'profile rbd pool=volumes, profile rbd pool=vms'
ceph auth get-or-create client.cinder-backup mon 'profile rbd' osd 'profile rbd pool=backups' mgr 'profile rbd pool=backups'
Which doesn't match with the existing permissions.
To modify the existing permissions I changed this to:
ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images' mgr 'profile rbd pool=volumes, profile rbd pool=vms'
ceph auth caps client.cinder-backup mon 'profile rbd' osd 'profile rbd pool=backups' mgr 'profile rbd pool=backups'
This however results in the following error message, for both commands:
Error EINVAL: mon capability parse failed, stopped at 'pool=images' of 'profile rbd pool=images'
Trying to create an additional client using the original commands from the documentation results in the same error messages.
How can I modify the permissions so cinder can connect properly?
Other clients seem to be working, gnocchi works, glance works, both with ceph backend enabled. It's just the cinder containers that keep restarting and reporting errors.
Versions involved:
- OpenStack: Train
- Ceph: Nautilus
- deployment: kolla-ansible, ceph-ansible
- both ceph and openstack are deployed in docker containers
Additional information from comments:
/etc/kolla/cinder-volume/cinder.conf
[rbd-1]
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
backend_host = rbd:volumes
rbd_pool = volumes
volume_backend_name = rbd-1
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = 4cc23cf2-2428-4efa-a0ac-757abaa70e01
I don't think the client requires mgr capabilities, at least I didn't set any in any of my settings. Your error message also seems to complain about exactly that, it stopped right after the osd caps (pool=images). The first configuration you posted seem fine though. Is the keyring file correct?
The keyring files seem correct. I just recreated them from
ceph auth getand redeployed them, but the content was already the same. The error messages persist.Can you share the cinder config (rbd part)? Have you checked if the network connection is properly configured?
I edited the rbd block from cinder.conf into the question. ceph.conf is absolutely identical for cinder and glance (where it works).