Ask Your Question
0

Ceilometer errors with RADOS Gateway - "NoSuchBucket"

asked 2017-09-11 01:42:59 -0600

jeremy.phillips gravatar image

We are running OpenStack Ocata on Ubuntu 16.04.3 (deployed using Fuel 11.0), with a Ceph Jewel (10.2.7) storage back-end. It is using RADOS Gateway as the object store in place of Swift, and that side of it works fine. The problem we have is getting Ceilometer to report on it correctly. Our ceilometer-polling.log file is filled with multiples of the following error every 10 minutes:

2017-09-11 02:45:53.324 21629 INFO swiftclient [-] REQ: curl -i http://10.4.78.35:8080/swift/v1/AUTH_f0db46a805284f1fb70a194744a78de6?format=json -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABZtfjXckJO..."
2017-09-11 02:45:53.324 21629 INFO swiftclient [-] RESP STATUS: 404 Not Found
2017-09-11 02:45:53.325 21629 INFO swiftclient [-] RESP HEADERS: {u'Content-Length': u'176', u'Accept-Ranges': u'bytes', u'Connection': u'close', u'X-Trans-Id': u'tx0000000000000000c6e1f-0059b5f8e1-3c90ef5-default', u'Date': u'Mon, 11 Sep 2017 02:45:53 GMT', u'Content-Type': u'application/json; charset=utf-8'}
2017-09-11 02:45:53.325 21629 INFO swiftclient [-] RESP BODY: {"Code":"NoSuchBucket","BucketName":"AUTH_f0db46a805284f1fb70a194744a78de6","RequestId":"tx0000000000000000c6e1f-0059b5f8e1-3c90ef5-default","HostId":"3c90ef5-default-default"}
2017-09-11 02:45:53.326 21629 WARNING ceilometer.objectstore.swift [-] Swift tenant id f0db46a805284f1fb70a194744a78de6 not found.

The problem seems to be that Ceilometer is treating the object store as Swift rather than RADOS Gateway (the warning message comes from ceilometer/objectstore/swify.py and not ceilometer/objectstore/rgw.py). Our Ceph configuration does not have rgw swift account in url = true set, so curl queries with AUTH_{tenant_id}in the URL will always fail (our preference is not to have to set this). I have been trying to configure Ceilometer to use RGW, with no luck so far. Following the instructions from https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage (https://docs.openstack.org/ceilometer...) :

  • I have set rgw enable usage log = true in ceph.conf.
  • I created a ceilometer account via radosgw-admin, with read/write access to buckets, metadata, usage and users.
  • I have set radosgw = object-store in ceilometer.conf, and set access_key and secret_key under [rgw_admin_credentials] to match the above ceilometer account.

Running ceilometer meter-list, I can see radosgw. meters appearing fine. But I'm still getting the above errors appearing. I don't know if this is a Ceph bug, a Ceilometer bug, or something else I've missed. Any suggestions?

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2017-09-12 19:22:58 -0600

jeremy.phillips gravatar image

In the end, we did end up re-creating the service endpoints for swift to include %(tenant_id)s at the end of the internal, public and admin URLs, as well as adding the lines rgw swift account in url = true and rgw keystone implicit tenants = false to our ceph.conf. After restarting the controllers, Ceilometer seems to work fine now.

edit flag offensive delete link more
0

answered 2018-06-20 01:42:56 -0600

I can confirm that your solution is one option to get it working. However, be aware that disabling http://docs.ceph.com/docs/mimic/radosgw/multitenancy/ (rgw keystone implicit tenants) will lead to seperate RGW users not being able to create buckets with the same name. Thus, for example, if you create a container "foo" with one OpenStack project and then try to create another container "foo" using another OpenStack project, this will fail. If you operate a shared cloud environment this is most probably not desired.

Unfortunately, the "multi-tenancy" feature changes the user naming scheme in RGW and this is what Ceilometer does not support. Thus, it gets an empty array back when querying the usage.

After a short debugging session, we managed to develop a patch for Ceilometer, adding an option to support implicit tenants in RGW. As of now, it has not been accepted, but if anyone finds this post in the future, here it is: https://review.openstack.org/#/c/574281/ The main change is to query the usage with uid being set to tenant_id + "$" + tenant_id instead of just tenant_id in ceilometer/objectstore/rgw_client.py.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2017-09-11 01:42:59 -0600

Seen: 435 times

Last updated: Sep 12 '17