keystone auth failure (is what it seems)
Up until approximately last week thursday, everything was normal.
However, now I cannot use any nova commands, ex:
\# nova list
ERROR: Unauthorized (HTTP 401)
In the api log, it just shows:
2014-07-28 13:56:22.707 8222 INFO nova.osapi_compute.wsgi.server [-] 10.0.25.2 "GET /v2/977cf273f3e546e1b5a70474bf81c266/servers/detail HTTP/1.1" status: 401 len: 466 time: 0.0328231
In the keystone log, it shows nothing, however when I run keystone, it shows a 401 on the console logs.
It's handing out tokens properly, but when anything tries to use a token, it fails..., from nova to any application written.
debug logs for nova-list:
\# nova --debug list
REQ: curl -i http://10.0.25.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "removed (as not really necessary)", "passwordCredentials": {"username": "removed", "password": "removed"}}}'
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.25.2
DEBUG (connectionpool:295) "POST /v2.0/tokens HTTP/1.1" 200 5152
RESP: [200] {'date': 'Mon, 28 Jul 2014 19:02:00 GMT', 'content-type': 'application/json', 'content-length': '5152', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-07-28T19:02:00.464555", "expires": "2014-07-29T19:02:00Z", "id": "MIIJKQYJKoZIhvcNAQcCoIIJGjCCCRYCAQExCTAHBgUrDgMCGjCCCAIGCSqGSIb3DQEHAaCCB-MEggfveyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWR
fYXQiOiAiMjAxNC0wNy0yOFQxOTowMjowMC40NjQ1NTUiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTI5VDE5OjAyOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIiIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2Ey
IiwgIm5hbWUiOiAic3lzIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuM
C4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAiaWQiOiAiNzJhMGEwOTYzMDY3NDEwODhkMWYwMGRhNGU5MTEyNzkiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3Ni92MS8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiJ9XSwgImVuZHBvaW50c1
9saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAiaWQiOiA
iZjdmZWFkOTM1NGQyNDc1NjgzNTI0MGVmMjE2MzBlYmYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6OTI5Mi92MSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4w
LjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJpZCI6ICI1MGM1ZWIxNjQ4Zjk0YTM3YmFkMDRkO
DI5NWVlYWVmZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4Nzc0L3YyLzJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaH
R0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogImNhNDk4ZmJjZTM4OTRmOGU4NDYxYjg0ZTYxNThjZDI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8
vMTAuMC4yNS4yOjg3NzMvc2VydmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJu
YWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo1MDAwL3YyLjAiLCAiaWQiOiAiM2IxYTYwNTViMzg1NGFmZmIxMjk3OGRiZTg5Nzk5NmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia
2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJzeXMiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogImNkM2FiNjU2ODg4YzRiMThiMDNhOGQ2MWJlN2Y3NDk4IiwgInJvbGVzIjogW3sibmFtZSI6ICJNZW1iZXIifV0sICJuYW1lIjogInN5cyJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgIn
JvbGVzIjogWyI5MjkzNzEwYTYyOWM0MDFhYWU1YjkyZTg2MGM5N2EzOSJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgFQ
OHOeYMkz+x5IVi+ewyFgxzEpTx5GBK2iJy+O0FQ1Eiymvu28h-iNoiD6cxJwzGIU46SOlW7cP6wrzeJQ3cWuxMQ8AyR-+5fWPaTuJWOX4jv1FEq-VPJOyeBLtuaIT96tCenZaeBGEe+gY7d77biNAA85m84sToJ1EsYzNad9y", "tenant": {"description": "", "enabled": true, "id": "2c59264fd8f
841cca8b4cdb038fd67a2", "name": "removed"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb038fd67a2", "region": "RegionOne", "internalURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb0
38fd67a2", "id": "72a0a096306741088d1f00da4e911279", "publicURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb038fd67a2"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://10.0.25.2:92
92/v1", "region": "RegionOne", "internalURL": "http://10.0.25.2:9292/v1", "id": "f7fead9354d24756835240ef21630ebf", "publicURL": "http://10.0.25.2:9292/v1"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"ad
minURL": "http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2", "region": "RegionOne", "internalURL": "http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2", "id": "50c5eb1648f94a37bad04d8295eeaefe", "publicURL": "http://10.0.
25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://10.0.25.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://10.0.25.2:8773
/services/Cloud", "id": "ca498fbce3894f8e8461b84e6158cd28", "publicURL": "http://10.0.25.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://10.0.25.2:35357/v2.0", "region":
"RegionOne", "internalURL": "http://10.0.25.2:5000/v2.0", "id": "3b1a6055b3854affb12978dbe897996b", "publicURL": "http://10.0.25.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "removed", "roles_links": [], "id": "cd3ab656888c4b18b03a8d61be7f7498", "roles": [{"name": "Member"}], "name": "removed"}, "metadata": {"is_admin": 0, "roles": ["9293710a629c401aae5b92e860c97a39"]}}}
REQ: curl -i http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2/servers/detail -X GET -H "X-Auth-Project-Id: removed" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: MIIJKQYJKoZIhvcNAQcCoIIJGjCCCRYCAQExCTAHBgUrDgMCGjCCCAIGCSqGSIb3DQEHAaCCB-MEggfveyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNy0yOFQxOTowMjowMC40NjQ1NTUiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTI5VDE5OjAyOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIiIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIiwgIm5hbWUiOiAic3lzIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAiaWQiOiAiNzJhMGEwOTYzMDY3NDEwODhkMWYwMGRhNGU5MTEyNzkiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3Ni92MS8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAiaWQiOiAiZjdmZWFkOTM1NGQyNDc1NjgzNTI0MGVmMjE2MzBlYmYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6OTI5Mi92MSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJpZCI6ICI1MGM1ZWIxNjQ4Zjk0YTM3YmFkMDRkODI5NWVlYWVmZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4Nzc0L3YyLzJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogImNhNDk4ZmJjZTM4OTRmOGU4NDYxYjg0ZTYxNThjZDI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzMvc2VydmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo1MDAwL3YyLjAiLCAiaWQiOiAiM2IxYTYwNTViMzg1NGFmZmIxMjk3OGRiZTg5Nzk5NmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJzeXMiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogImNkM2FiNjU2ODg4YzRiMThiMDNhOGQ2MWJlN2Y3NDk4IiwgInJvbGVzIjogW3sibmFtZSI6ICJNZW1iZXIifV0sICJuYW1lIjogInN5cyJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5MjkzNzEwYTYyOWM0MDFhYWU1YjkyZTg2MGM5N2EzOSJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgFQOHOeYMkz+x5IVi+ewyFgxzEpTx5GBK2iJy+O0FQ1Eiymvu28h-iNoiD6cxJwzGIU46SOlW7cP6wrzeJQ3cWuxMQ8AyR-+5fWPaTuJWOX4jv1FEq-VPJOyeBLtuaIT96tCenZaeBGEe+gY7d77biNAA85m84sToJ1EsYzNad9y"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.25.2
DEBUG (connectionpool:295) "GET /v2/2c59264fd8f841cca8b4cdb038fd67a2/servers/detail HTTP/1.1" 401 276
RESP: [401] {'date': 'Mon, 28 Jul 2014 19 ...