Hello everyone,

I kept running openstack grizzly for over one year until last week. Users report me that they couldn't delete instances from dashboard. After check on the controller node, I found all commands (except keystone) returned the Unauthorized 401 error. Since all services (cinder/nova/glance/quantum) have the same issue, I think it might caused by keystone. I also checked the user, role assignment for each tenant which are all correct. please find my novarc file, nova list debug out put and keystone debug log below. Is there other tests I could do to trace the root cause?

Thanks in advance,

novarc file:

export OS_REGION_NAME=Jinan

export OS_PASSWORD=xxxxxxx

export OS_AUTH_URL=http://192.168.99.1:5000/v2.0

export OS_USERNAME=admin

export OS_TENANT_NAME=admin

Nova debug output

root@controller:/etc/keystone# nova --debug list

REQ: curl -i http://127.0.0.1:35357/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "rsp", "passwordCredentials": {"username": "service", "password": "xxxxxx"}}}'

INFO (connectionpool:191) Starting new HTTP connection (1): 127.0.0.1 DEBUG (connectionpool:283) "POST /v2.0/tokens HTTP/1.1" 200 6034 RESP: [200] {'date': 'Tue, 15 Apr 2014 05:54:09 GMT', 'content-type': 'application/json', 'content-length': '6034', 'vary': 'X-Auth-Token'} RESP BODY: {"access": {"token": {"issued_at": "2014-04-15T05:54:09.073077", "expires": "2014-04-16T05:54:09Z", "id": "MIIKowYJKoZIhvcNAQcCoIIKlDCCCpACAQExCTAHBgUrDgMCGjCCCXwGCSqGSIb3DQEHAaCCCW0EgglpeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNC0xNVQwNTo1NDowOS4wNzMwNzciLCAiZXhwaXJlcyI6ICIyMDE0LTA0LTE2VDA1OjU0OjA5WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2VzIE1hbmFnZW1lbnQgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEiLCAibmFtZSI6ICJyc3AifX0sICJzZXJ2aWNlQ2F0YWxvZyI6IFt7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6ODc3NC92Mi8xNjkzMzc2YTg2OWI0YzhkYjRhODYzNTlkNDg5OWFlYSIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo4Nzc0L3YyLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIiwgImlkIjogIjFlZjVlNjk3NjYwODQ5MDU5OWQ3YTdmNDU0NzRlZmJiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMi4yMC43MTo4Nzc0L3YyLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyIsICJpZCI6ICIzMzI0MGUxNTA4YjY0MTkxYjZkOWIxYjdlMjM4ZjI0ZiIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5Njk2LyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJuZXR3b3JrIiwgIm5hbWUiOiAicXVhbnR1bSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6OTI5Mi92MiIsICJyZWdpb24iOiAiSmluYW4iLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo5MjkyL3YyIiwgImlkIjogIjAzMGRiYzk5NjQ1NDRlNTRiMDBhMTc2ODM1YTlhOWNiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMi4yMC43MTo5MjkyL3YyIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImltYWdlIiwgIm5hbWUiOiAiZ2xhbmNlIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMjo4Nzc2L3YxLzE2OTMzNzZhODY5YjRjOGRiNGE4NjM1OWQ0ODk5YWVhIiwgInJlZ2lvbiI6ICJKaW5hbiIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4yOjg3NzYvdjEvMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEiLCAiaWQiOiAiMGZiMTQzYmE4MzgwNDhjYTkwNzRlMDE2MGY0NmI0YTUiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4yLjIwLjcwOjg3NzYvdjEvMTY5MzM3NmE4NjliNGM4ZGI0YTg2MzU5ZDQ4OTlhZWEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjguOTkuMTo4NzczL3NlcnZpY2VzL0FkbWluIiwgInJlZ2lvbiI6ICJKaW5hbiIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjg3NzMvc2VydmljZXMvQ2xvdWQiLCAiaWQiOiAiNDM5OTE3YWE0NjIxNGQxZjhiMTA0N2ViYmM5ZmE1NGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6ODc3My9zZXJ2aWNlcy9DbG91ZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJlYzIiLCAibmFtZSI6ICJlYzIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogIkppbmFuIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY4Ljk5LjE6NTAwMC92Mi4wIiwgImlkIjogIjJhZjEyNmM3Y2UwMjRiN2JiMjJhYzQ1YTZhNzBiNjQyIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OC45OS4xOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidXNlcm5hbWUiOiAic2VydmljZSIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiYTcxMGJjNmIxNzQyNGJkZmFlMmRlZGIyYTRiOTc0OWQiLCAicm9sZXMiOiBbeyJuYW1lIjogIl9tZW1iZXJfIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic2VydmljZSJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5ZmUyZmY5ZWU0Mzg0YjE4OTRhOTA4NzhkM2U5MmJhYiIsICJiYWVkMGVhNWU0YmE0OWRlOTc0MDE2YTZiMTZiOGQ4NiJdfX19MYH-MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgD3-EOkD-sSApthkBg7Eu1E+RXNPfZaOjpAPm+Bf7mqavKItFtCwQ-5KIu8rVPVpn1ZvlEsieAdeX6NETiyjQ6harSUD1M8jMhGTa7pX2ud3jWXav-qs-7ie7++VgPPPgeie3aGlyXhjum8U-H8l+QNZWq1WVj6OkkRwjo2JmxSB", "tenant": {"description": "Services Management Tenant", "enabled": true, "id": "1693376a869b4c8db4a86359d4899aea", "name": "rsp"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.99.1:8774/v2/1693376a869b4c8db4a86359d4899aea", "region": "Jinan", "internalURL": "http://192.168.99.1:8774/v2/1693376a869b4c8db4a86359d4899aea", "id": "1ef5e6976608490599d7a7f45474efbb", "publicURL": "http://10.2.20.71:8774/v2/1693376a869b4c8db4a86359d4899aea"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.99.1:9696/", "region": "Jinan", "internalURL": "http://192.168.99.1:9696/", "id": "33240e1508b64191b6d9b1b7e238f24f", "publicURL": "http://192.168.99.1:9696/"}], "endpoints_links": [], "type": "network", "name": "quantum"}, {"endpoints": [{"adminURL": "http://192.168.99.1:9292/v2", "region": "Jinan", "internalURL": "http://192.168.99.1:9292/v2", "id": "030dbc9964544e54b00a176835a9a9cb", "publicURL": "http://10.2.20.71:9292/v2"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.99.2:8776/v1/1693376a869b4c8db4a86359d4899aea", "region": "Jinan", "internalURL": "http://192.168.99.2:8776/v1/1693376a869b4c8db4a86359d4899aea", "id": "0fb143ba838048ca9074e0160f46b4a5", "publicURL": "http://10.2.20.70:8776/v1/1693376a869b4c8db4a86359d4899aea"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.99.1:8773/services/Admin", "region": "Jinan", "internalURL": "http://192.168.99.1:8773/services/Cloud", "id": "439917aa46214d1f8b1047ebbc9fa54f", "publicURL": "http://192.168.99.1:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://192.168.99.1:35357/v2.0", "region": "Jinan", "internalURL": "http://192.168.99.1:5000/v2.0", "id": "2af126c7ce024b7bb22ac45a6a70b642", "publicURL": "http://192.168.99.1:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "service", "roles_links": [], "id": "a710bc6b17424bdfae2dedb2a4b9749d", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "service"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "baed0ea5e4ba49de974016a6b16b8d86"]}}}

REQ: curl -i http://10.2.20.71:8774/v2/1693376a869b4c8db4a86359d4899aea/servers/detail (http://10.2.20.71:8774/v2/1693376a869...) -X GET ... (more)