Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

asked 2014-11-01 15:00:09 -0500

don gravatar image

periodic failure to authenticate services in juno

In general my Juno installation (on Ubuntu 14.10) is working well. However, about once every day, i get a 401 unauthorized for some service.Once in this state, it stays there.

For example, it might be the neutronclient inside nova. Or, right now, it is cinder.

The case I have right now, in the horizon dashboard, I get "unable to retrieve volume limit information". If I run 'cinder list' as the user who logged into glance, it works.

the traceback in horizon is:

Traceback (most recent call last):
  File "./openstack_dashboard/usage/base.py", line 184, in get_cinder_limits
    self.limits.update(api.cinder.tenant_absolute_limits(self.request))
  File "/usr/lib/python2.7/dist-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "./openstack_dashboard/api/cinder.py", line 482, in tenant_absolute_limits
    limits = cinderclient(request).limits.get().absolute
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/limits.py", line 92, in get
    return self._get("/limits", "limits")
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 149, in _get
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 302, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 263, in _cs_request
    self.authenticate()
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 411, in authenticate
    auth_url = self._v1_auth(auth_url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 429, in _v1_auth
    resp, body = self.request(url, 'GET', headers=headers)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 252, in request
    raise exceptions.from_response(resp, body)
Unauthorized: Unauthorized (HTTP 401)

And the associated log in keystone is:

2014-11-01 19:51:44.454 13427 DEBUG keystone.middleware.core [-] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:280
2014-11-01 19:51:44.457 13427 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /usr/lib/python2.7/dist-packages/keystone/common/wsgi.py:191
2014-11-01 19:51:44.458 13427 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:revocation_list() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:55
2014-11-01 19:51:44.459 13427 DEBUG keystone.common.controller [-] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:60
2014-11-01 19:51:44.459 13427 DEBUG keystone.policy.backends.rules [-] enforce identity:revocation_list: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:100
2014-11-01 19:51:44.460 13427 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:155
2014-11-01 19:51:44.488 13427 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/revoked HTTP/1.1" 200 2866 0.059528
2014-11-01 19:51:44.806 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.808 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.810 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.008629
2014-11-01 19:51:44.838 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.839 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.840 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.006543

If i restart keystone and horizon, the problem will vanish again for a while (1+ days). Some token is timing out, and it doesn't seem to know how to get a new one.

Although the problem is cinder right now, it can equally be any service (glance, neutron etc).

In cinder.conf I have:

[keystone_authtoken]
auth_uri = https://MYHOST:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = CINDER_PASSWORD
admin_user = cinder
admin_tenant_name = service
cafile =
auth_protocol = https
auth_port = 35357
auth_host = MYHOST

can someone suggest what I have wrong?

periodic failure to authenticate services in juno

In general my Juno installation (on Ubuntu 14.10) is working well. However, about once every day, i get a 401 unauthorized for some service.Once in this state, it stays there.

For example, it might be the neutronclient inside nova. Or, right now, it is cinder.

The case I have right now, in the horizon dashboard, I get "unable to retrieve volume limit information". If I run 'cinder list' as the user who logged into glance, it works.

the traceback in horizon is:

Traceback (most recent call last):
  File "./openstack_dashboard/usage/base.py", line 184, in get_cinder_limits
    self.limits.update(api.cinder.tenant_absolute_limits(self.request))
  File "/usr/lib/python2.7/dist-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "./openstack_dashboard/api/cinder.py", line 482, in tenant_absolute_limits
    limits = cinderclient(request).limits.get().absolute
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/limits.py", line 92, in get
    return self._get("/limits", "limits")
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 149, in _get
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 302, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 263, in _cs_request
    self.authenticate()
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 411, in authenticate
    auth_url = self._v1_auth(auth_url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 429, in _v1_auth
    resp, body = self.request(url, 'GET', headers=headers)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 252, in request
    raise exceptions.from_response(resp, body)
Unauthorized: Unauthorized (HTTP 401)

And the associated log in keystone is:

2014-11-01 19:51:44.454 13427 DEBUG keystone.middleware.core [-] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:280
2014-11-01 19:51:44.457 13427 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /usr/lib/python2.7/dist-packages/keystone/common/wsgi.py:191
2014-11-01 19:51:44.458 13427 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:revocation_list() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:55
2014-11-01 19:51:44.459 13427 DEBUG keystone.common.controller [-] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:60
2014-11-01 19:51:44.459 13427 DEBUG keystone.policy.backends.rules [-] enforce identity:revocation_list: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:100
2014-11-01 19:51:44.460 13427 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:155
2014-11-01 19:51:44.488 13427 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/revoked HTTP/1.1" 200 2866 0.059528
2014-11-01 19:51:44.806 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.808 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.810 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.008629
2014-11-01 19:51:44.838 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.839 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.840 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.006543

If i restart keystone and horizon, and/or memcache, the problem stays. if i restart cinder, the problem will vanish again for a while (1+ days). Some token is timing out, and it doesn't seem to know how to get a new one.

Although the problem is cinder right now, it can equally be any service (glance, neutron etc).

In cinder.conf I have:

[keystone_authtoken]
auth_uri = https://MYHOST:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = CINDER_PASSWORD
admin_user = cinder
admin_tenant_name = service
cafile =
auth_protocol = https
auth_port = 35357
auth_host = MYHOST

can someone suggest what I have wrong?

periodic failure to authenticate services in juno

In general my Juno installation (on Ubuntu 14.10) is working well. However, about once every day, i get a 401 unauthorized for some service.Once in this state, it stays there.

For example, it might be the neutronclient inside nova. Or, right now, it is cinder.

The case I have right now, in the horizon dashboard, I get "unable to retrieve volume limit information". If I run 'cinder list' as the user who logged into glance, it works.

the traceback in horizon is:

Traceback (most recent call last):
  File "./openstack_dashboard/usage/base.py", line 184, in get_cinder_limits
    self.limits.update(api.cinder.tenant_absolute_limits(self.request))
  File "/usr/lib/python2.7/dist-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "./openstack_dashboard/api/cinder.py", line 482, in tenant_absolute_limits
    limits = cinderclient(request).limits.get().absolute
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/limits.py", line 92, in get
    return self._get("/limits", "limits")
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 149, in _get
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 302, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 263, in _cs_request
    self.authenticate()
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 411, in authenticate
    auth_url = self._v1_auth(auth_url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 429, in _v1_auth
    resp, body = self.request(url, 'GET', headers=headers)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 252, in request
    raise exceptions.from_response(resp, body)
Unauthorized: Unauthorized (HTTP 401)

And the associated log in keystone is:

2014-11-01 19:51:44.454 13427 DEBUG keystone.middleware.core [-] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:280
2014-11-01 19:51:44.457 13427 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /usr/lib/python2.7/dist-packages/keystone/common/wsgi.py:191
2014-11-01 19:51:44.458 13427 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:revocation_list() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:55
2014-11-01 19:51:44.459 13427 DEBUG keystone.common.controller [-] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:60
2014-11-01 19:51:44.459 13427 DEBUG keystone.policy.backends.rules [-] enforce identity:revocation_list: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:100
2014-11-01 19:51:44.460 13427 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:155
2014-11-01 19:51:44.488 13427 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/revoked HTTP/1.1" 200 2866 0.059528
2014-11-01 19:51:44.806 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.808 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.810 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.008629
2014-11-01 19:51:44.838 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.839 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.840 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.006543

The log for cinder says:

2014-11-01 20:00:56.397 13458 WARNING keystonemiddleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication. (Disable debug mode to suppress these details.)', u'code': 401, u'title': u'Unauthorized'}}
2014-11-01 20:00:56.398 13458 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-11-01 20:00:56.404 13458 WARNING keystonemiddleware.auth_token [-] Unable to find authentication token in headers

If i restart keystone and/or memcache, the problem stays. if i restart cinder, the problem will vanish again for a while (1+ days). Some token is timing out, and it doesn't seem to know how to get a new one.

Although the problem is cinder right now, it can equally be any service (glance, neutron etc).

In cinder.conf I have:

[keystone_authtoken]
auth_uri = https://MYHOST:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = CINDER_PASSWORD
admin_user = cinder
admin_tenant_name = service
cafile =
auth_protocol = https
auth_port = 35357
auth_host = MYHOST

can someone suggest what I have wrong?

periodic failure to authenticate services in juno

In general my Juno installation (on Ubuntu 14.10) is working well. However, about once every day, i get a 401 unauthorized for some service.Once in this state, it stays there.

For example, it might be the neutronclient inside nova. Or, right now, it is cinder.

The case I have right now, in the horizon dashboard, I get "unable to retrieve volume limit information". If I run 'cinder list' as the user who logged into glance, it works.

the traceback in horizon is:

Traceback (most recent call last):
  File "./openstack_dashboard/usage/base.py", line 184, in get_cinder_limits
    self.limits.update(api.cinder.tenant_absolute_limits(self.request))
  File "/usr/lib/python2.7/dist-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "./openstack_dashboard/api/cinder.py", line 482, in tenant_absolute_limits
    limits = cinderclient(request).limits.get().absolute
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/limits.py", line 92, in get
    return self._get("/limits", "limits")
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 149, in _get
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 302, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 263, in _cs_request
    self.authenticate()
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 411, in authenticate
    auth_url = self._v1_auth(auth_url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 429, in _v1_auth
    resp, body = self.request(url, 'GET', headers=headers)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 252, in request
    raise exceptions.from_response(resp, body)
Unauthorized: Unauthorized (HTTP 401)

And the associated log in keystone is:

2014-11-01 19:51:44.454 13427 DEBUG keystone.middleware.core [-] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:280
2014-11-01 19:51:44.457 13427 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /usr/lib/python2.7/dist-packages/keystone/common/wsgi.py:191
2014-11-01 19:51:44.458 13427 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:revocation_list() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:55
2014-11-01 19:51:44.459 13427 DEBUG keystone.common.controller [-] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:60
2014-11-01 19:51:44.459 13427 DEBUG keystone.policy.backends.rules [-] enforce identity:revocation_list: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'b9b6f116169145629090c92dd968fca4', 'roles': [u'admin', u'_member_'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=fLEFkiAsSXyENjGs0M-5AA, audit_chain_id=fLEFkiAsSXyENjGs0M-5AA) at 0x7f0adfbcad08>, 'project_id': u'96f1c1e2c5e34c3991c509f7d2cb15bf', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:100
2014-11-01 19:51:44.460 13427 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:155
2014-11-01 19:51:44.488 13427 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/revoked HTTP/1.1" 200 2866 0.059528
2014-11-01 19:51:44.806 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.808 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.810 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.008629
2014-11-01 19:51:44.838 13429 WARNING keystone.middleware.core [-] RBAC: Invalid token(core) token
2014-11-01 19:51:44.839 13429 WARNING keystone.common.wsgi [-] The request you have made requires authentication. (Disable debug mode to suppress these details.)
2014-11-01 19:51:44.840 13429 INFO eventlet.wsgi.server [-] 192.168.10.66 - - [01/Nov/2014 19:51:44] "GET /v2.0/tokens/fdb058bd60c24bde890495c28ac2ab26 HTTP/1.1" 401 395 0.006543

The log for cinder says:

2014-11-01 20:00:56.397 13458 WARNING keystonemiddleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication. (Disable debug mode to suppress these details.)', u'code': 401, u'title': u'Unauthorized'}}
2014-11-01 20:00:56.398 13458 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-11-01 20:00:56.404 13458 WARNING keystonemiddleware.auth_token [-] Unable to find authentication token in headers

If i restart keystone and/or memcache, the problem stays. if i restart cinder, the problem will vanish again for a while (1+ days). Some token is timing out, and it doesn't seem to know how to get a new one.

Although the problem is cinder right now, it can equally be any service (glance, neutron etc).

In cinder.conf I have:

[keystone_authtoken]
auth_uri = https://MYHOST:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = CINDER_PASSWORD
admin_user = cinder
admin_tenant_name = service
cafile =
auth_protocol = https
auth_port = 35357
auth_host = MYHOST

If i do token-get:

$ OS_TENANT_NAME=service OS_USERNAME=cinder OS_PASSWORD='CINDER_PASSWORD' keystone token-get
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2014-11-11T20:22:33Z       |
|     id    | 05eb84a543cf45dd8765f9cdd39f16bb |
| tenant_id | 96f1c1e2c5e34c3991c509f7d2cb15bf |
|  user_id  | 48f89c2a1362427abc7c60cc1647a603 |
+-----------+----------------------------------+

can someone suggest what I have wrong?