Ask Your Question
1

Grizzly Keystone authentication issue

asked 2014-04-15 01:23:18 -0600

wcy1323 gravatar image

updated 2014-04-15 04:33:36 -0600

Hello everyone,

I kept running openstack grizzly for over one year until last week. Users report me that they couldn't delete instances from dashboard. After check on the controller node, I found all commands (except keystone) returned the Unauthorized 401 error. Since all services (cinder/nova/glance/quantum) have the same issue, I think it might caused by keystone. I also checked the user, role assignment for each tenant which are all correct. please find my novarc file, nova list debug out put and keystone debug log below. Is there other tests I could do to trace the root cause?

Thanks in advance,

novarc file:

export OS_REGION_NAME=Jinan

export OS_PASSWORD=xxxxxxx

export OS_AUTH_URL=http://192.168.99.1:5000/v2.0

export OS_USERNAME=admin

export OS_TENANT_NAME=admin

Nova debug output

root@controller:/etc/keystone# nova --debug list

REQ: curl -i http://127.0.0.1:35357/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "rsp", "passwordCredentials": {"username": "service", "password": "xxxxxx"}}}'

INFO (connectionpool:191) Starting new HTTP connection (1): 127.0.0.1 DEBUG (connectionpool:283) "POST /v2.0/tokens HTTP/1.1" 200 6034 RESP: [200] {'date': 'Tue, 15 Apr 2014 05:54:09 GMT', 'content-type': 'application/json', 'content-length': '6034', 'vary': 'X-Auth-Token'} RESP BODY: {"access": {"token": {"issued_at": "2014-04-15T05:54:09.073077", "expires": "2014-04-16T05:54:09Z", "id": "MIIKowYJKoZIhvcNAQcCoIIKlDCCCpACAQExCTAHBgUrDgMCGjCCCXwGCSqGSIb3DQEHAaCCCW0EgglpeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNC0xNVQwNTo1NDowOS4wNzMwNzciLCAiZXhwaXJlcyI6ICIyMDE0LTA0LTE2VDA1OjU0OjA5WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2VzIE1hbmFnZW1lbnQgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEiLCAibmFtZSI6ICJyc3AifX0sICJzZXJ2aWNlQ2F0YWxvZyI6IFt7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6ODc3NC92Mi8xNjkzMzc2YTg2OWI0YzhkYjRhODYzNTlkNDg5OWFlYSIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo4Nzc0L3YyLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIiwgImlkIjogIjFlZjVlNjk3NjYwODQ5MDU5OWQ3YTdmNDU0NzRlZmJiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMi4yMC43MTo4Nzc0L3YyLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyIsICJpZCI6ICIzMzI0MGUxNTA4YjY0MTkxYjZkOWIxYjdlMjM4ZjI0ZiIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJuZXR3b3JrIiwgIm5hbWUiOiAicXVhbnR1bSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6OTI5Mi92MiIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5MjkyL3YyIiwgImlkIjogIjAzMGRiYzk5NjQ1NDRlNTRiMDBhMTc2ODM1YTlhOWNiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMi4yMC43MTo5MjkyL3YyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImltYWdlIiwgIm5hbWUiOiAiZ2xhbmNlIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMjo4Nzc2L3YxLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIiwgInJlZ2lvbiI6ICJKaW5hbiIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4yOjg3NzYvdjEvMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEiLCAiaWQiOiAiMGZiMTQzYmE4MzgwNDhjYTkwNzRlMDE2MGY0NmI0YTUiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4yLjIwLjcwOjg3NzYvdjEvMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJKaW5hbiIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjg3NzMvc2VydmljZXMvQ2xvdWQiLCAiaWQiOiAiNDM5OTE3YWE0NjIxNGQxZjhiMTA0N2ViYmM5ZmE1NGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6ODc3My9zZXJ2aWNlcy9DbG91ZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJlYzIiLCAibmFtZSI6ICJlYzIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogIkppbmFuIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6NTAwMC92Mi4wIiwgImlkIjogIjJhZjEyNmM3Y2UwMjRiN2JiMjJhYzQ1YTZhNzBiNjQyIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidXNlcm5hbWUiOiAic2VydmljZSIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiYTcxMGJjNmIxNzQyNGJkZmFlMmRlZGIyYTRiOTc0OWQiLCAicm9sZXMiOiBbeyJuYW1lIjogIl9tZW1iZXJfIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic2VydmljZSJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5ZmUyZmY5ZWU0Mzg0YjE4OTRhOTA4NzhkM2U5MmJhYiIsICJiYWVkMGVhNWU0YmE0OWRlOTc0MDE2YTZiMTZiOGQ4NiJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgD3-EOkD-sSApthkBg7Eu1E+RXNPfZaOjpAPm+Bf7mqavKItFtCwQ-5KIu8rVPVpn1ZvlEsieAdeX6NETiyjQ6harSUD1M8jMhGTa7pX2ud3jWXav-qs-7ie7++VgPPPgeie3aGlyXhjum8U-H8l+QNZWq1WVj6OkkRwjo2JmxSB", "tenant": {"description": "Services Management Tenant", "enabled": true, "id": "1693376a869b4c8db4a86359d4899aea", "name": "rsp"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.99.1:8774/v2/1693376a869b4c8db4a86359d4899aea", "region": "Jinan", "internalURL": "http://192.168.99.1:8774/v2/1693376a869b4c8db4a86359d4899aea", "id": "1ef5e6976608490599d7a7f45474efbb", "publicURL": "http://10.2.20.71:8774/v2/1693376a869b4c8db4a86359d4899aea"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.99.1:9696/", "region": "Jinan", "internalURL": "http://192.168.99.1:9696/", "id": "33240e1508b64191b6d9b1b7e238f24f", "publicURL": "http://192.168.99.1:9696/"}], "endpoints_links": [], "type": "network", "name": "quantum"}, {"endpoints": [{"adminURL": "http://192.168.99.1:9292/v2", "region": "Jinan", "internalURL": "http://192.168.99.1:9292/v2", "id": "030dbc9964544e54b00a176835a9a9cb", "publicURL": "http://10.2.20.71:9292/v2"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.99.2:8776/v1/1693376a869b4c8db4a86359d4899aea", "region": "Jinan", "internalURL": "http://192.168.99.2:8776/v1/1693376a869b4c8db4a86359d4899aea", "id": "0fb143ba838048ca9074e0160f46b4a5", "publicURL": "http://10.2.20.70:8776/v1/1693376a869b4c8db4a86359d4899aea"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.99.1:8773/services/Admin", "region": "Jinan", "internalURL": "http://192.168.99.1:8773/services/Cloud", "id": "439917aa46214d1f8b1047ebbc9fa54f", "publicURL": "http://192.168.99.1:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://192.168.99.1:35357/v2.0", "region": "Jinan", "internalURL": "http://192.168.99.1:5000/v2.0", "id": "2af126c7ce024b7bb22ac45a6a70b642", "publicURL": "http://192.168.99.1:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "service", "roles_links": [], "id": "a710bc6b17424bdfae2dedb2a4b9749d", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "service"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "baed0ea5e4ba49de974016a6b16b8d86"]}}}

REQ: curl -i http://10.2.20.71:8774/v2/1693376a869b4c8db4a86359d4899aea/servers/detail (http://10.2.20.71:8774/v2/1693376a869...) -X GET ... (more)

edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted
0

answered 2014-04-28 08:12:35 -0600

I am having the same issue. I think the issue is because the certificate used to create token's has expired. I used keystone-manage‘s pki_setup to generate a new one. Now horizon seems to be working ok but nova and glance are still giving me 401's.

edit flag offensive delete link more

Comments

The final step to fix the issue was to remove all the cached certificates for all the services. In my case: cacert.pem revoked.pem signing_cert.pem

For each nova, quantum, and glance

cyberoblivion gravatar imagecyberoblivion ( 2014-04-28 10:58:52 -0600 )edit

Hi, Where there any other steps than just removing the .pem files ? I have the exact same problem, but removing the certs didn't fix the issue.

Thanks

vthm gravatar imagevthm ( 2014-05-08 22:03:12 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-04-15 01:23:18 -0600

Seen: 297 times

Last updated: Apr 28 '14