Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

keystone auth failure (is what it seems)

Up until approximately last week thursday, everything was normal.

However, now I cannot use any nova commands, ex:

\# nova list

ERROR: Unauthorized (HTTP 401)

In the api log, it just shows:

2014-07-28 13:56:22.707 8222 INFO nova.osapi_compute.wsgi.server [-] 10.0.25.2 "GET /v2/977cf273f3e546e1b5a70474bf81c266/servers/detail HTTP/1.1" status: 401 len: 466 time: 0.0328231

In the keystone log, it shows nothing, however when I run keystone, it shows a 401 on the console logs.

It's handing out tokens properly, but when anything tries to use a token, it fails..., from nova to any application written.

debug logs for nova-list:

\# nova --debug list

REQ: curl -i http://10.0.25.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "removed (as not really necessary)", "passwordCredentials": {"username": "removed", "password": "removed"}}}'

INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.25.2
DEBUG (connectionpool:295) "POST /v2.0/tokens HTTP/1.1" 200 5152
RESP: [200] {'date': 'Mon, 28 Jul 2014 19:02:00 GMT', 'content-type': 'application/json', 'content-length': '5152', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-07-28T19:02:00.464555", "expires": "2014-07-29T19:02:00Z", "id": "MIIJKQYJKoZIhvcNAQcCoIIJGjCCCRYCAQExCTAHBgUrDgMCGjCCCAIGCSqGSIb3DQEHAaCCB-MEggfveyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWR
fYXQiOiAiMjAxNC0wNy0yOFQxOTowMjowMC40NjQ1NTUiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTI5VDE5OjAyOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIiIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2Ey
IiwgIm5hbWUiOiAic3lzIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuM
C4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAiaWQiOiAiNzJhMGEwOTYzMDY3NDEwODhkMWYwMGRhNGU5MTEyNzkiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3Ni92MS8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiJ9XSwgImVuZHBvaW50c1
9saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAiaWQiOiA
iZjdmZWFkOTM1NGQyNDc1NjgzNTI0MGVmMjE2MzBlYmYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6OTI5Mi92MSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4w
LjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJpZCI6ICI1MGM1ZWIxNjQ4Zjk0YTM3YmFkMDRkO
DI5NWVlYWVmZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4Nzc0L3YyLzJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaH
R0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogImNhNDk4ZmJjZTM4OTRmOGU4NDYxYjg0ZTYxNThjZDI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8
vMTAuMC4yNS4yOjg3NzMvc2VydmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJu
YWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo1MDAwL3YyLjAiLCAiaWQiOiAiM2IxYTYwNTViMzg1NGFmZmIxMjk3OGRiZTg5Nzk5NmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia
2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJzeXMiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogImNkM2FiNjU2ODg4YzRiMThiMDNhOGQ2MWJlN2Y3NDk4IiwgInJvbGVzIjogW3sibmFtZSI6ICJNZW1iZXIifV0sICJuYW1lIjogInN5cyJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgIn
JvbGVzIjogWyI5MjkzNzEwYTYyOWM0MDFhYWU1YjkyZTg2MGM5N2EzOSJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgFQ
OHOeYMkz+x5IVi+ewyFgxzEpTx5GBK2iJy+O0FQ1Eiymvu28h-iNoiD6cxJwzGIU46SOlW7cP6wrzeJQ3cWuxMQ8AyR-+5fWPaTuJWOX4jv1FEq-VPJOyeBLtuaIT96tCenZaeBGEe+gY7d77biNAA85m84sToJ1EsYzNad9y", "tenant": {"description": "", "enabled": true, "id": "2c59264fd8f
841cca8b4cdb038fd67a2", "name": "removed"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb038fd67a2", "region": "RegionOne", "internalURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb0
38fd67a2", "id": "72a0a096306741088d1f00da4e911279", "publicURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb038fd67a2"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://10.0.25.2:92
92/v1", "region": "RegionOne", "internalURL": "http://10.0.25.2:9292/v1", "id": "f7fead9354d24756835240ef21630ebf", "publicURL": "http://10.0.25.2:9292/v1"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"ad
minURL": "http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2", "region": "RegionOne", "internalURL": "http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2", "id": "50c5eb1648f94a37bad04d8295eeaefe", "publicURL": "http://10.0.
25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://10.0.25.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://10.0.25.2:8773
/services/Cloud", "id": "ca498fbce3894f8e8461b84e6158cd28", "publicURL": "http://10.0.25.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://10.0.25.2:35357/v2.0", "region":
 "RegionOne", "internalURL": "http://10.0.25.2:5000/v2.0", "id": "3b1a6055b3854affb12978dbe897996b", "publicURL": "http://10.0.25.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "removed", "roles_links": [], "id": "cd3ab656888c4b18b03a8d61be7f7498", "roles": [{"name": "Member"}], "name": "removed"}, "metadata": {"is_admin": 0, "roles": ["9293710a629c401aae5b92e860c97a39"]}}}


REQ: curl -i http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2/servers/detail -X GET -H "X-Auth-Project-Id: removed" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: MIIJKQYJKoZIhvcNAQcCoIIJGjCCCRYCAQExCTAHBgUrDgMCGjCCCAIGCSqGSIb3DQEHAaCCB-MEggfveyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNy0yOFQxOTowMjowMC40NjQ1NTUiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTI5VDE5OjAyOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIiIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIiwgIm5hbWUiOiAic3lzIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAiaWQiOiAiNzJhMGEwOTYzMDY3NDEwODhkMWYwMGRhNGU5MTEyNzkiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3Ni92MS8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAiaWQiOiAiZjdmZWFkOTM1NGQyNDc1NjgzNTI0MGVmMjE2MzBlYmYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6OTI5Mi92MSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJpZCI6ICI1MGM1ZWIxNjQ4Zjk0YTM3YmFkMDRkODI5NWVlYWVmZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4Nzc0L3YyLzJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogImNhNDk4ZmJjZTM4OTRmOGU4NDYxYjg0ZTYxNThjZDI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzMvc2VydmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo1MDAwL3YyLjAiLCAiaWQiOiAiM2IxYTYwNTViMzg1NGFmZmIxMjk3OGRiZTg5Nzk5NmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJzeXMiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogImNkM2FiNjU2ODg4YzRiMThiMDNhOGQ2MWJlN2Y3NDk4IiwgInJvbGVzIjogW3sibmFtZSI6ICJNZW1iZXIifV0sICJuYW1lIjogInN5cyJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5MjkzNzEwYTYyOWM0MDFhYWU1YjkyZTg2MGM5N2EzOSJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgFQOHOeYMkz+x5IVi+ewyFgxzEpTx5GBK2iJy+O0FQ1Eiymvu28h-iNoiD6cxJwzGIU46SOlW7cP6wrzeJQ3cWuxMQ8AyR-+5fWPaTuJWOX4jv1FEq-VPJOyeBLtuaIT96tCenZaeBGEe+gY7d77biNAA85m84sToJ1EsYzNad9y" 

INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.25.2
DEBUG (connectionpool:295) "GET /v2/2c59264fd8f841cca8b4cdb038fd67a2/servers/detail HTTP/1.1" 401 276
RESP: [401] {'date': 'Mon, 28 Jul 2014 19:02:00 GMT', 'content-length': '276', 'content-type': 'text/plain; charset=UTF-8', 'www-authenticate': "Keystone uri='http://10.0.25.2:5000/v2.0/'"}
RESP BODY: 401 Unauthorized

This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.

 Authentication required

Nothing has been changed before the issue started, it just... stopped working as far as I know.