Unable to flush expired tokens - token.ibd

asked 2017-08-31 05:32:28 -0600

Rajesh Ramachandran gravatar image

I have a RH openstack setup (undercloud and overcloud) with a single director and 3 controllers in HA mode and remaining computes.

I was trying to backup the undercloud database and observed that the DB size was more.

token.ibd is occupying more space in the disk :

# du -sch /var/lib/mysql/keystone/token.ibd
57G     /var/lib/mysql/keystone/token.ibd
57G     total

There is a cron job to execute keystone-manage token_flush command, but it looks like the cron job is not working as expected.

I tried manually running the “keystone-manage token_flush” command, it just hangs and doesn’t help. I don’t see any obvious errors from mariadb log.

If the size/number of expired tokens are high, does that cause the token_flush command to fail to handle ? If so, how do we deal with flushing the expired keystone tokens ?

edit retag flag offensive close merge delete


Ideas: Check the db - are there any expired tokens? Check the keystone log - warnings or errors? Also in the log, check for "Flush expired tokens process started", "Total expired tokens removed:", and if DEBUG logging is enabled, "Token expiration batch size".

Bernd Bausch gravatar imageBernd Bausch ( 2017-08-31 22:34:53 -0600 )edit

Perhaps it doesn't hang, just takes a very long time. More ideas: Any disk activity? The top command shows that it uses CPU?

Bernd Bausch gravatar imageBernd Bausch ( 2017-08-31 22:35:28 -0600 )edit

Thanks Bernd for your suggestions. As you said, it took long time because of large number of expired tokens. All good now. I also reduced the size of token.ibd by "optimize table token;"

Rajesh Ramachandran gravatar imageRajesh Ramachandran ( 2017-09-03 23:31:51 -0600 )edit