Grizzly Keystone authentication issue
Hello everyone,
I kept running openstack grizzly for over one year until last week. Users report me that they couldn't delete instances from dashboard. After check on the controller node, I found all commands (except keystone) returned the Unauthorized 401 error. Since all services (cinder/nova/glance/quantum) have the same issue, I think it might caused by keystone. I also checked the user, role assignment for each tenant which are all correct. please find my novarc file, nova list debug out put and keystone debug log below. Is there other tests I could do to trace the root cause?
Thanks in advance,
novarc file:
export OS_REGION_NAME=Jinan
export OS_PASSWORD=xxxxxxx
export OS_AUTH_URL=http://192.168.99.1:5000/v2.0
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
Nova debug output
root@controller:/etc/keystone# nova --debug list
REQ: curl -i http://127.0.0.1:35357/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "rsp", "passwordCredentials": {"username": "service", "password": "xxxxxx"}}}'
INFO (connectionpool:191) Starting new HTTP connection (1): 127.0.0.1 DEBUG (connectionpool:283) "POST /v2.0/tokens HTTP/1.1" 200 6034 RESP: [200] {'date': 'Tue, 15 Apr 2014 05:54:09 GMT', 'content-type': 'application/json', 'content-length': '6034', 'vary': 'X-Auth-Token'} RESP BODY: {"access": {"token": {"issued_at": "2014-04-15T05:54:09.073077", "expires": "2014-04-16T05:54:09Z", "id": "MIIKowYJKoZIhvcNAQcCoIIKlDCCCpACAQExCTAHBgUrDgMCGjCCCXwGCSqGSIb3DQEHAaCCCW0EgglpeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNC0xNVQwNTo1NDowOS4wNzMwNzciLCAiZXhwaXJlcyI6ICIyMDE0LTA0LTE2VDA1OjU0OjA5WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2VzIE1hbmFnZW1lbnQgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEiLCAibmFtZSI6ICJyc3AifX0sICJzZXJ2aWNlQ2F0YWxvZyI6IFt7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6ODc3NC92Mi8xNjkzMzc2YTg2OWI0YzhkYjRhODYzNTlkNDg5OWFlYSIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo4Nzc0L3YyLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIiwgImlkIjogIjFlZjVlNjk3NjYwODQ5MDU5OWQ3YTdmNDU0NzRlZmJiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMi4yMC43MTo4Nzc0L3YyLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyIsICJpZCI6ICIzMzI0MGUxNTA4YjY0MTkxYjZkOWIxYjdlMjM4ZjI0ZiIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJuZXR3b3JrIiwgIm5hbWUiOiAicXVhbnR1bSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6OTI5Mi92MiIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5MjkyL3YyIiwgImlkIjogIjAzMGRiYzk5NjQ1NDRlNTRiMDBhMTc2ODM1YTlhOWNiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMi4yMC43MTo5MjkyL3YyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImltYWdlIiwgIm5hbWUiOiAiZ2xhbmNlIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMjo4Nzc2L3YxLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIiwgInJlZ2lvbiI6ICJKaW5hbiIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4yOjg3NzYvdjEvMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEiLCAiaWQiOiAiMGZiMTQzYmE4MzgwNDhjYTkwNzRlMDE2MGY0NmI0YTUiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4yLjIwLjcwOjg3NzYvdjEvMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJKaW5hbiIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjg3NzMvc2VydmljZXMvQ2xvdWQiLCAiaWQiOiAiNDM5OTE3YWE0NjIxNGQxZjhiMTA0N2ViYmM5ZmE1NGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6ODc3My9zZXJ2aWNlcy9DbG91ZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJlYzIiLCAibmFtZSI6ICJlYzIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogIkppbmFuIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6NTAwMC92Mi4wIiwgImlkIjogIjJhZjEyNmM3Y2UwMjRiN2JiMjJhYzQ1YTZhNzBiNjQyIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidXNlcm5hbWUiOiAic2VydmljZSIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiYTcxMGJjNmIxNzQyNGJkZmFlMmRlZGIyYTRiOTc0OWQiLCAicm9sZXMiOiBbeyJuYW1lIjogIl9tZW1iZXJfIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic2VydmljZSJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5ZmUyZmY5ZWU0Mzg0YjE4OTRhOTA4NzhkM2U5MmJhYiIsICJiYWVkMGVhNWU0YmE0OWRlOTc0MDE2YTZiMTZiOGQ4NiJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgD3-EOkD-sSApthkBg7Eu1E+RXNPfZaOjpAPm+Bf7mqavKItFtCwQ-5KIu8rVPVpn1ZvlEsieAdeX6NETiyjQ6harSUD1M8jMhGTa7pX2ud3jWXav-qs-7ie7++VgPPPgeie3aGlyXhjum8U-H8l+QNZWq1WVj6OkkRwjo2JmxSB", "tenant": {"description": "Services Management Tenant", "enabled": true, "id": "1693376a869b4c8db4a86359d4899aea", "name": "rsp"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.99.1:8774/v2/1693376a869b4c8db4a86359d4899aea", "region": "Jinan", "internalURL": "http://192.168.99.1:8774/v2/1693376a869b4c8db4a86359d4899aea", "id": "1ef5e6976608490599d7a7f45474efbb", "publicURL": "http://10.2.20.71:8774/v2/1693376a869b4c8db4a86359d4899aea"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.99.1:9696/", "region": "Jinan", "internalURL": "http://192.168.99.1:9696/", "id": "33240e1508b64191b6d9b1b7e238f24f", "publicURL": "http://192.168.99.1:9696/"}], "endpoints_links": [], "type": "network", "name": "quantum"}, {"endpoints": [{"adminURL": "http://192.168.99.1:9292/v2", "region": "Jinan", "internalURL": "http://192.168.99.1:9292/v2", "id": "030dbc9964544e54b00a176835a9a9cb", "publicURL": "http://10.2.20.71:9292/v2"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.99.2:8776/v1/1693376a869b4c8db4a86359d4899aea", "region": "Jinan", "internalURL": "http://192.168.99.2:8776/v1/1693376a869b4c8db4a86359d4899aea", "id": "0fb143ba838048ca9074e0160f46b4a5", "publicURL": "http://10.2.20.70:8776/v1/1693376a869b4c8db4a86359d4899aea"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.99.1:8773/services/Admin", "region": "Jinan", "internalURL": "http://192.168.99.1:8773/services/Cloud", "id": "439917aa46214d1f8b1047ebbc9fa54f", "publicURL": "http://192.168.99.1:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://192.168.99.1:35357/v2.0", "region": "Jinan", "internalURL": "http://192.168.99.1:5000/v2.0", "id": "2af126c7ce024b7bb22ac45a6a70b642", "publicURL": "http://192.168.99.1:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "service", "roles_links": [], "id": "a710bc6b17424bdfae2dedb2a4b9749d", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "service"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "baed0ea5e4ba49de974016a6b16b8d86"]}}}
REQ: curl -i http://10.2.20.71:8774/v2/1693376a869b4c8db4a86359d4899aea/servers/detail (http://10.2.20.71:8774/v2/1693376a869...) -X GET ...