How to workaround ceph reshard log spam (ceph bug 20289)?

asked 2019-08-02 03:56:27 -0500

Aphid gravatar image

Also see: https://tracker.ceph.com/issues/20289. I'm seeing lots of this pattern in my ceph-client.rgw.localhost log:

[timestamp] [code] -1 ERROR: failed to list reshard log entries, oid=reshard.0000000000

I'm having the same problem. The Ceph hosts are writing ~20GB of that to their disks each day. Logrotate is clearing it but I fear for the lifespan of my disks.

The user that submitted the bug posted a solution, but they were too terse in their description for it to be useful. Exactly how does one change the 'radosgw user' permissions? (And I mean this question in both possible meanings: i.e.

1) what commands do you execute?

2) which permissions need changing?

For my case, printing the permissions using ceph auth get client.rgw.localhost results in the following output, for which I'm presuming rgw.localhost is the problem user, as the log file that's getting spammed is named after this user.

[client.rgw.localhost]
        key = [crypto key here]
        caps mgr = "allow r"
        caps mon = "allow rw"
        caps osd = "allow rwx"

By my limited understanding, that means it has full permissions on all the osds, independent of pool... why am I still seeing the reshard log entry spam, is some other thing necessary?

Some extra information: I have the following 'pools' that are not manually created (e.g. 'rgw' pools), listed using ceph osd lspools. Notice that I do not have a default.rgw.reshard pool:

1 .rgw.root,
2 default.rgw.control,
3 default.rgw.data.root,
4 default.rgw.gc,
5 default.rgw.log,
7 default.rgw.users.uid,
edit retag flag offensive close merge delete