Unable to flush expired tokens - token.ibd
I have a RH openstack setup (undercloud and overcloud) with a single director and 3 controllers in HA mode and remaining computes.
I was trying to backup the undercloud database and observed that the DB size was more.
token.ibd is occupying more space in the disk :
# du -sch /var/lib/mysql/keystone/token.ibd
57G /var/lib/mysql/keystone/token.ibd
57G total
There is a cron job to execute keystone-manage token_flush command, but it looks like the cron job is not working as expected.
I tried manually running the “keystone-manage token_flush” command, it just hangs and doesn’t help. I don’t see any obvious errors from mariadb log.
If the size/number of expired tokens are high, does that cause the token_flush command to fail to handle ? If so, how do we deal with flushing the expired keystone tokens ?
Ideas: Check the db - are there any expired tokens? Check the keystone log - warnings or errors? Also in the log, check for "Flush expired tokens process started", "Total expired tokens removed:", and if DEBUG logging is enabled, "Token expiration batch size".
Perhaps it doesn't hang, just takes a very long time. More ideas: Any disk activity? The top command shows that it uses CPU?
Thanks Bernd for your suggestions. As you said, it took long time because of large number of expired tokens. All good now. I also reduced the size of token.ibd by "optimize table token;"