ObjectNotFound: [errno 2] error connecting to the cluster (Cinder+Ceph))

asked 2020-01-31 01:34:00 -0500

randomname gravatar image

Its something wrong with rbd driver: Volume driver RBDDriver not initialized

what could be the reason for this?

cinder.conf

    [DEFAULT]
transport_url=rabbit://amqp_user:Abcd1234@ip:5672/
auth_strategy = keystone
my_ip = ip
enabled_backends=ceph
debug = True
#glance_api_version = 2

...

    [ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = 5
rbd_user = cinder
rbd_cluster_name = ceph
rbd_secret_uuid = 11eac55-e4fe-49d0-8bee-a37b8caef68f
rbd_keyring_conf = /etc/ceph/ceph.client.cinder.keyring
vlume_clear = none

ceph.conf

[global]
fsid = 302b9536-c79d-4839-b5a7-33fe70ee272d
mon_initial_members = s1, s2, s3
mon_host = ip1, ip2, ip3
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_pool_default_size = 2
public_network = 111.111.11.11/25
[client.images]
keyring = /etc/ceph/ceph.client.images.keyring
[client.cinder]
keyring = /etc/ceph/ceph.cinder.volumes.keyring

volume.log

2020-01-31 16:18:39.585 2272390 DEBUG cinder.utils [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:802
2020-01-31 16:18:39.585 2272390 DEBUG cinder.utils [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Have been at this for 0.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:804
2020-01-31 16:18:39.586 2272390 DEBUG cinder.utils [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:796
2020-01-31 16:18:49.587 2272390 DEBUG cinder.volume.drivers.rbd [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=5). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:402
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Error connecting to ceph cluster.: ObjectNotFound: [errno 2] error connecting to the cluster
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd Traceback (most recent call last):
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 415, in _do_conn
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd     client.connect()
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd   File "rados.pyx", line 893, in rados.Rados.connect (/builddir/build/BUILD/ceph-14.2.1/build/src/pybind/rados/pyrex/rados.c:9834)
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd ObjectNotFound: [errno 2] error connecting to the cluster
2020-01-31 16:18:49.655 2272390 ERROR cinder.volume.drivers.rbd 
2020-01-31 16:18:49.657 2272390 DEBUG cinder.utils [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:802
2020-01-31 16:18:49.657 2272390 DEBUG cinder.utils [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Have been at this for 10.145 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:804
2020-01-31 16:18:49.658 2272390 DEBUG cinder.utils [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:796
2020-01-31 16:19:09.658 2272390 DEBUG cinder.volume.drivers.rbd [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=5). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:402
2020-01-31 16:19:09.731 2272390 ERROR cinder.volume.drivers.rbd [req-d21c2a90-66ae-40c6-addb-bfaadd8245f5 - - - - -] Error ...
(more)
edit retag flag offensive close merge delete

Comments

I don't know much about connecting Cinder to Ceph, but this line in the backend description looks incorrect:

vlume_clear = none
Bernd Bausch gravatar imageBernd Bausch ( 2020-01-31 04:29:12 -0500 )edit

I fixed it, it doesn't affect the result. I tried the same without this line, it was added only because the ticket with a similar problem advised to prescribe it.

randomname gravatar imagerandomname ( 2020-02-02 18:22:10 -0500 )edit

Except for the obvious typo (see Bernd's comment) I would try to connect manually first to confirm the permissions. Run rbd -p volumes --id cinder -k /etc/ceph/ceph.client.cinder.keyring ls from the cinder node to validate your credentials. If that works please update the question or comment.

eblock gravatar imageeblock ( 2020-02-03 02:00:04 -0500 )edit

Did you put ceph keyring into /etc/ceph folder and enbled rdb module by command "modprobe rbd" ?

tienbm90 gravatar imagetienbm90 ( 2020-02-03 22:32:59 -0500 )edit