periodic failure to authenticate services in juno

asked 2014-11-01 15:00:09 -0600

don gravatar image

updated 2014-11-01 15:23:22 -0600

In general my Juno installation (on Ubuntu 14.10) is working well. However, about once every day, i get a 401 unauthorized for some service.Once in this state, it stays there.

For example, it might be the neutronclient inside nova. Or, right now, it is cinder.

The case I have right now, in the horizon dashboard, I get "unable to retrieve volume limit information". If I run 'cinder list' as the user who logged into glance, it works.

the traceback in horizon is:

Traceback (most recent call last):
  File "./openstack_dashboard/usage/base.py", line 184, in get_cinder_limits
    self.limits.update(api.cinder.tenant_absolute_limits(self.request))
  File "/usr/lib/python2.7/dist-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "./openstack_dashboard/api/cinder.py", line 482, in tenant_absolute_limits
    limits = cinderclient(request).limits.get().absolute
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/limits.py", line 92, in get
    return self._get("/limits", "limits")
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 149, in _get
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 302, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 263, in _cs_request
    self.authenticate()
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 411, in authenticate
    auth_url = self._v1_auth(auth_url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 429, in _v1_auth
    resp, body = self.request(url, 'GET', headers=headers)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 252, in request
    raise exceptions.from_response(resp, body)
Unauthorized: Unauthorized (HTTP 401)

And the associated log in keystone is:

2014-11-01 19:51:44.454 13427 DEBUG keystone.middleware.core [-] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:280
2014-11-01 19:51:44.457 13427 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /usr/lib/python2.7/dist-packages/keystone/common/wsgi.py:191
2014-11-01 19:51:44.458 13427 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:revocation_list() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:55
2014-11-01 19:51:44.459 13427 DEBUG keystone.common.controller [-] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:60
2014-11-01 19:51:44.459 13427 DEBUG keystone.policy.backends.rules [-] enforce identity:revocation_list: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:100
2014-11-01 19:51:44.460 13427 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:155
2014-11-01 19:51:44.488 13427 INFO eventlet ...
(more)
edit retag flag offensive close merge delete

Comments

51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.008629
2014-11-01 19:51:44.838 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token

Can you please get the details about the token fdb058bd60c24bde890495c28ac2ab26 from the token table.

Haneef Ali gravatar imageHaneef Ali ( 2014-11-01 22:38:17 -0600 )edit

I'm not sure how to do that. the token table in mysql is empty (memcached I guess).

I'm using the default provider (uuid), and memcache driver

i've now restarted so i'm in not in a state that is broken.

don gravatar imagedon ( 2014-11-02 05:59:20 -0600 )edit

Hi, I have the same issue, did you resolve it?

Cristiano gravatar imageCristiano ( 2017-04-06 07:49:36 -0600 )edit