one user's ec2api credentials stopped working

asked 2013-10-18 16:04:05 -0500

jproulx gravatar image

updated 2013-10-25 17:16:31 -0500

smaffulli gravatar image

I have a user whose ec2api credentials stopped working recently. I'm told they worked about a week ago which is well after the last system level changes. System is Grizzly using sql backend for ec2 tokens (though memcache for "normal" tokens).

I can see that he can use the osapi and dashboard. I can also see that freshly downloaded ec2 credentials from the dashboard fail on a client system where my ec2api credentials and other users in the same project do work.

nova-api.log on the server side says:

2013-10-18 09:30:15.142 5338 ERROR nova.api.ec2 [-] Unauthorized: Failure communicating with keystone

keystone.log says:

2013-10-18 09:30:15  WARNING [keystone.common.wsgi] Unable to add token user list.

The user tokens EC2_ACCESS_KEY and EC2_SECRET_KEY in the downloaded ec2rc.sh match what is in the ec2_credential table of the keystone db:

mysql> select user.name,ec2_credential.access,ec2_credential.secret from ec2_credential,user where ec2_credential.user_id=user.id and user.name='mybrokenuser';
+----------------+----------------------------------+----------------------------------+
| name           | access                           | secret                           |
+----------------+----------------------------------+----------------------------------+
| mybrokenuser   | $EC2_ACCESS_KEY value            | $EC2_SECRET_KEY value            |
+----------------+----------------------------------+----------------------------------+

I removed the row with the problematic ec2_credential then created a new one with keystone ec2-credentials-create, but still seemed to have the same problem. In the interest of getting the user working again to meet a deadline I took the expedient step of deleting and recreating his user account. In retro spect I probably should have renamed the broken account to preserve state for investigation.

Unfortunately this means there's not much chance of getting any more debuging information that what I've presented here, but if anyone can deduce a possible cause from this information a a similar exerience I'd very much like to know what went wrong.

edit retag flag offensive close delete