Ask Your Question
1

keystone auth failure (is what it seems)

asked 2014-07-28 14:09:35 -0600

akiSa gravatar image

Up until approximately last week thursday, everything was normal.

However, now I cannot use any nova commands, ex:

\# nova list

ERROR: Unauthorized (HTTP 401)

In the api log, it just shows:

2014-07-28 13:56:22.707 8222 INFO nova.osapi_compute.wsgi.server [-] 10.0.25.2 "GET /v2/977cf273f3e546e1b5a70474bf81c266/servers/detail HTTP/1.1" status: 401 len: 466 time: 0.0328231

In the keystone log, it shows nothing, however when I run keystone, it shows a 401 on the console logs.

It's handing out tokens properly, but when anything tries to use a token, it fails..., from nova to any application written.

debug logs for nova-list:

\# nova --debug list

REQ: curl -i http://10.0.25.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "removed (as not really necessary)", "passwordCredentials": {"username": "removed", "password": "removed"}}}'

INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.25.2
DEBUG (connectionpool:295) "POST /v2.0/tokens HTTP/1.1" 200 5152
RESP: [200] {'date': 'Mon, 28 Jul 2014 19:02:00 GMT', 'content-type': 'application/json', 'content-length': '5152', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-07-28T19:02:00.464555", "expires": "2014-07-29T19:02:00Z", "id": "MIIJKQYJKoZIhvcNAQcCoIIJGjCCCRYCAQExCTAHBgUrDgMCGjCCCAIGCSqGSIb3DQEHAaCCB-MEggfveyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWR
fYXQiOiAiMjAxNC0wNy0yOFQxOTowMjowMC40NjQ1NTUiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTI5VDE5OjAyOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIiIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2Ey
IiwgIm5hbWUiOiAic3lzIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuM
C4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAiaWQiOiAiNzJhMGEwOTYzMDY3NDEwODhkMWYwMGRhNGU5MTEyNzkiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3Ni92MS8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiJ9XSwgImVuZHBvaW50c1
9saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAiaWQiOiA
iZjdmZWFkOTM1NGQyNDc1NjgzNTI0MGVmMjE2MzBlYmYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6OTI5Mi92MSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4w
LjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJpZCI6ICI1MGM1ZWIxNjQ4Zjk0YTM3YmFkMDRkO
DI5NWVlYWVmZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4Nzc0L3YyLzJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaH
R0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogImNhNDk4ZmJjZTM4OTRmOGU4NDYxYjg0ZTYxNThjZDI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8
vMTAuMC4yNS4yOjg3NzMvc2VydmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJu
YWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo1MDAwL3YyLjAiLCAiaWQiOiAiM2IxYTYwNTViMzg1NGFmZmIxMjk3OGRiZTg5Nzk5NmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia
2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJzeXMiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogImNkM2FiNjU2ODg4YzRiMThiMDNhOGQ2MWJlN2Y3NDk4IiwgInJvbGVzIjogW3sibmFtZSI6ICJNZW1iZXIifV0sICJuYW1lIjogInN5cyJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgIn
JvbGVzIjogWyI5MjkzNzEwYTYyOWM0MDFhYWU1YjkyZTg2MGM5N2EzOSJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgFQ
OHOeYMkz+x5IVi+ewyFgxzEpTx5GBK2iJy+O0FQ1Eiymvu28h-iNoiD6cxJwzGIU46SOlW7cP6wrzeJQ3cWuxMQ8AyR-+5fWPaTuJWOX4jv1FEq-VPJOyeBLtuaIT96tCenZaeBGEe+gY7d77biNAA85m84sToJ1EsYzNad9y", "tenant": {"description": "", "enabled": true, "id": "2c59264fd8f
841cca8b4cdb038fd67a2", "name": "removed"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb038fd67a2", "region": "RegionOne", "internalURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb0
38fd67a2", "id": "72a0a096306741088d1f00da4e911279", "publicURL": "http://10.0.25.2:8776/v1/2c59264fd8f841cca8b4cdb038fd67a2"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://10.0.25.2:92
92/v1", "region": "RegionOne", "internalURL": "http://10.0.25.2:9292/v1", "id": "f7fead9354d24756835240ef21630ebf", "publicURL": "http://10.0.25.2:9292/v1"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"ad
minURL": "http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2", "region": "RegionOne", "internalURL": "http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2", "id": "50c5eb1648f94a37bad04d8295eeaefe", "publicURL": "http://10.0.
25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://10.0.25.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://10.0.25.2:8773
/services/Cloud", "id": "ca498fbce3894f8e8461b84e6158cd28", "publicURL": "http://10.0.25.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://10.0.25.2:35357/v2.0", "region":
 "RegionOne", "internalURL": "http://10.0.25.2:5000/v2.0", "id": "3b1a6055b3854affb12978dbe897996b", "publicURL": "http://10.0.25.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "removed", "roles_links": [], "id": "cd3ab656888c4b18b03a8d61be7f7498", "roles": [{"name": "Member"}], "name": "removed"}, "metadata": {"is_admin": 0, "roles": ["9293710a629c401aae5b92e860c97a39"]}}}


REQ: curl -i http://10.0.25.2:8774/v2/2c59264fd8f841cca8b4cdb038fd67a2/servers/detail -X GET -H "X-Auth-Project-Id: removed" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: MIIJKQYJKoZIhvcNAQcCoIIJGjCCCRYCAQExCTAHBgUrDgMCGjCCCAIGCSqGSIb3DQEHAaCCB-MEggfveyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNy0yOFQxOTowMjowMC40NjQ1NTUiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTI5VDE5OjAyOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIiIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIiwgIm5hbWUiOiAic3lzIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzYvdjEvMmM1OTI2NGZkOGY4NDFjY2E4YjRjZGIwMzhmZDY3YTIiLCAiaWQiOiAiNzJhMGEwOTYzMDY3NDEwODhkMWYwMGRhNGU5MTEyNzkiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3Ni92MS8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjkyOTIvdjEiLCAiaWQiOiAiZjdmZWFkOTM1NGQyNDc1NjgzNTI0MGVmMjE2MzBlYmYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6OTI5Mi92MSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6ODc3NC92Mi8yYzU5MjY0ZmQ4Zjg0MWNjYThiNGNkYjAzOGZkNjdhMiIsICJpZCI6ICI1MGM1ZWIxNjQ4Zjk0YTM3YmFkMDRkODI5NWVlYWVmZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4Nzc0L3YyLzJjNTkyNjRmZDhmODQxY2NhOGI0Y2RiMDM4ZmQ2N2EyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogImNhNDk4ZmJjZTM4OTRmOGU4NDYxYjg0ZTYxNThjZDI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMC4yNS4yOjg3NzMvc2VydmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjAuMjUuMjozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMjUuMjo1MDAwL3YyLjAiLCAiaWQiOiAiM2IxYTYwNTViMzg1NGFmZmIxMjk3OGRiZTg5Nzk5NmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4wLjI1LjI6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJzeXMiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogImNkM2FiNjU2ODg4YzRiMThiMDNhOGQ2MWJlN2Y3NDk4IiwgInJvbGVzIjogW3sibmFtZSI6ICJNZW1iZXIifV0sICJuYW1lIjogInN5cyJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5MjkzNzEwYTYyOWM0MDFhYWU1YjkyZTg2MGM5N2EzOSJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgFQOHOeYMkz+x5IVi+ewyFgxzEpTx5GBK2iJy+O0FQ1Eiymvu28h-iNoiD6cxJwzGIU46SOlW7cP6wrzeJQ3cWuxMQ8AyR-+5fWPaTuJWOX4jv1FEq-VPJOyeBLtuaIT96tCenZaeBGEe+gY7d77biNAA85m84sToJ1EsYzNad9y" 

INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.25.2
DEBUG (connectionpool:295) "GET /v2/2c59264fd8f841cca8b4cdb038fd67a2/servers/detail HTTP/1.1" 401 276
RESP: [401] {'date': 'Mon, 28 Jul 2014 19 ...
(more)
edit retag flag offensive close merge delete
add a comment

1 answer

Sort by ยป oldest newest most voted
1

answered 2014-07-28 15:59:35 -0600

rooter gravatar image

updated 2014-07-28 16:02:37 -0600

When nova-api receives a API request (e.g. server list), it validates the token passed to from the client it against keystone. But in order to do that, it has to first authenticate against keystone using it's own credentials 401 Unauthorised most likely means, that nova-api was unable to authenticate with keystone (and this is "unauthorised" to access the URL used for validating tokens).

In other words, this is most likely caused by credentials in /etc/nova/nova.conf of the node hosting the nova-api being out of sync with the actual credentials assigned to the nova user.

[root@openstack1 ~]# cat /etc/nova/nova.conf  | grep ^admin_
admin_user=nova
admin_password=zT8SsObhAHdqqgZPc
admin_tenant_name=service

Check those against what you think is set for the nova user in openstack.

You can easily confirm this theory. Issuing the following command (with XXXs replaced with whatever was returned with the command above) should work on a healthy OpenStack deployment. If it doesn't, this means nova's credentials are off somewhere (either in nova.conf or in keystone db).

curl -i -X POST http://10.0.25.2:5000/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "XXXX", "passwordCredentials": {"username": "XXX", "password": "XXXX"}}}'
edit flag offensive delete link more

Comments

Sorry for the late response.. Firstly, I don't see how the credentials could've changed in the keystone db, because nothing has changed within the nova conf. However, for some strange reason, I'm able to auth by using the horizon web ui, but not with nova/curl req's... which is why I'm baffled.

akiSa gravatar imageakiSa ( 2014-07-29 10:23:17 -0600 )edit
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2014-07-28 14:09:35 -0600

Seen: 2,666 times

Last updated: Jul 28 '14

Related questions

ProgrammingError: (ProgrammingError) (1146, "Table 'nova.services' doesn't exist")

ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.106:5672 is unreachable: [Errno 111] ECONNREFUSED.

412 Precondition Failed on swift -a

Openstack VM Created with Error Status

heat template - create instance, attach cinder volume, use it as LVM

keystone via httpd and remote_user

RabbitMQ is down

could not telnet to keystone port from another machine

nova list Status ERROR [closed]

Keystone connection establishment error