Ask Your Question

jeremy.phillips's profile - activity

2018-05-21 11:40:01 -0600 received badge  Famous Question (source)
2017-12-20 02:37:53 -0600 received badge  Notable Question (source)
2017-10-31 07:42:53 -0600 received badge  Popular Question (source)
2017-09-12 19:22:58 -0600 answered a question Ceilometer errors with RADOS Gateway - "NoSuchBucket"

In the end, we did end up re-creating the service endpoints for swift to include %(tenant_id)s at the end of the internal, public and admin URLs, as well as adding the lines rgw swift account in url = true and rgw keystone implicit tenants = false to our ceph.conf. After restarting the controllers, Ceilometer seems to work fine now.

2017-09-11 15:35:23 -0600 asked a question Ceilometer errors with RADOS Gateway - "NoSuchBucket"

We are running OpenStack Ocata on Ubuntu 16.04.3 (deployed using Fuel 11.0), with a Ceph Jewel (10.2.7) storage back-end. It is using RADOS Gateway as the object store in place of Swift, and that side of it works fine. The problem we have is getting Ceilometer to report on it correctly. Our ceilometer-polling.log file is filled with multiples of the following error every 10 minutes:

2017-09-11 02:45:53.324 21629 INFO swiftclient [-] REQ: curl -i http://10.4.78.35:8080/swift/v1/AUTH_f0db46a805284f1fb70a194744a78de6?format=json -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABZtfjXckJO..."
2017-09-11 02:45:53.324 21629 INFO swiftclient [-] RESP STATUS: 404 Not Found
2017-09-11 02:45:53.325 21629 INFO swiftclient [-] RESP HEADERS: {u'Content-Length': u'176', u'Accept-Ranges': u'bytes', u'Connection': u'close', u'X-Trans-Id': u'tx0000000000000000c6e1f-0059b5f8e1-3c90ef5-default', u'Date': u'Mon, 11 Sep 2017 02:45:53 GMT', u'Content-Type': u'application/json; charset=utf-8'}
2017-09-11 02:45:53.325 21629 INFO swiftclient [-] RESP BODY: {"Code":"NoSuchBucket","BucketName":"AUTH_f0db46a805284f1fb70a194744a78de6","RequestId":"tx0000000000000000c6e1f-0059b5f8e1-3c90ef5-default","HostId":"3c90ef5-default-default"}
2017-09-11 02:45:53.326 21629 WARNING ceilometer.objectstore.swift [-] Swift tenant id f0db46a805284f1fb70a194744a78de6 not found.

The problem seems to be that Ceilometer is treating the object store as Swift rather than RADOS Gateway (the warning message comes from ceilometer/objectstore/swify.py and not ceilometer/objectstore/rgw.py). Our Ceph configuration does not have rgw swift account in url = true set, so curl queries with AUTH_{tenant_id}in the URL will always fail (our preference is not to have to set this). I have been trying to configure Ceilometer to use RGW, with no luck so far. Following the instructions from https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage (https://docs.openstack.org/ceilometer...) :

  • I have set rgw enable usage log = true in ceph.conf.
  • I created a ceilometer account via radosgw-admin, with read/write access to buckets, metadata, usage and users.
  • I have set radosgw = object-store in ceilometer.conf, and set access_key and secret_key under [rgw_admin_credentials] to match the above ceilometer account.

Running ceilometer meter-list, I can see radosgw. meters appearing fine. But I'm still getting the above errors appearing. I don't know if this is a Ceph bug, a Ceilometer bug, or something else I've missed. Any suggestions?