Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Keystone 'user' commands fail with `HTTP 500 Internal Server Error`

I'm running through a Havana install on Ubuntu 12.04 and I am unable to modify users. Most of the keystone commands work, but any command that require modifying the db (such as user-delete or user-role-add) fail with Internal Server Error (HTTP 500).

I'm using MariaDB as the backend. No errors in syslog or query-log.

I do see a lot of rollback queries in the query-log, but I assume that's normal?

Keystone 'user' commands fail with `HTTP 500 Internal Server Error`

I'm running through a Havana install on Ubuntu 12.04 and I am unable to modify users. Most of the keystone commands work, but any command that require modifying the db (such as user-delete or user-role-add) fail with Internal Server Error (HTTP 500).

I'm using MariaDB as the backend. No errors in syslog or query-log.

I do see a lot of rollback queries in the query-log, but I assume that's normal?

 keystone --debug user-role-add --user=glance --tenant=service --role=admin
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://10.204.112.58:35357/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "openstack"}}}'
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 None
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"access": {"token": {"expires": "2014-04-08T21:16:59Z", "id": "7fcb3d62905347ad8160d244c06fdd19", "tenant": {"description": "Admin Tenant", "enabled": true, "id": "654a42abfa354f1bac6abb82b8a62b88", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.204.112.58:35357/v2.0", "region": "regionOne", "internalURL": "http://10.204.112.58:5000/v2.0", "publicURL": "http://10.204.112.58:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "7a7616fbbcf64f5cbe9be8f0c60498eb", "roles": [{"id": "324aa65afb4e4249a0522a8e117b604e", "name": "admin"}], "name": "admin"}}}

DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.204.112.58:35357/v2.0/users/glance -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/users/glance HTTP/1.1" 404 None
DEBUG:keystoneclient.session:RESP: [404] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"error": {"message": "Could not find user: glance", "code": 404, "title": "Not Found"}}

DEBUG:keystoneclient.session:Request returned failure status: 404
DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.204.112.58:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/users HTTP/1.1" 200 None
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"users": [{"name": "glance", "enabled": true, "email": "glance@localhost", "id": "38528bc3020c4d9db0bdfa63247f744b", "tenantId": null}, {"name": "admin", "enabled": true, "email": "root@localhost", "id": "7a7616fbbcf64f5cbe9be8f0c60498eb", "tenantId": null}]}

DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.204.112.58:35357/v2.0/OS-KSADM/roles/admin -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/OS-KSADM/roles/admin HTTP/1.1" 404 None
DEBUG:keystoneclient.session:RESP: [404] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"error": {"message": "Could not find role: admin", "code": 404, "title": "Not Found"}}

DEBUG:keystoneclient.session:Request returned failure status: 404
DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.204.112.58:35357/v2.0/OS-KSADM/roles -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/OS-KSADM/roles HTTP/1.1" 200 None
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"roles": [{"id": "324aa65afb4e4249a0522a8e117b604e", "name": "admin"}]}

DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.204.112.58:35357/v2.0/tenants/service -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/tenants/service HTTP/1.1" 404 None
DEBUG:keystoneclient.session:RESP: [404] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"error": {"message": "Could not find tenant: service", "code": 404, "title": "Not Found"}}

DEBUG:keystoneclient.session:Request returned failure status: 404
DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.204.112.58:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/tenants HTTP/1.1" 200 None
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'transfer-encoding': 'chunked', 'vary': 'X-Auth-Token', 'content-type': 'application/json'})
RESP BODY: {"tenants_links": [], "tenants": [{"enabled": true, "description": "Admin Tenant", "name": "admin", "id": "654a42abfa354f1bac6abb82b8a62b88"}, {"enabled": true, "description": "Service Tenant", "name": "service", "id": "a6beb75468d94818988bdd97335f3c1b"}]}

DEBUG:iso8601.iso8601:Parsed 2014-04-08T21:16:59Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'21', 'daydash': u'08', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'59', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'04', 'day': None, 'minute': u'16'} with default timezone <iso8601.iso8601.Utc object at 0x1f97b10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'04' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 4 for 'month' with default 4
DEBUG:iso8601.iso8601:Got u'08' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'day' with default 8
DEBUG:iso8601.iso8601:Got u'21' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'16' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'59' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X PUT http://10.204.112.58:35357/v2.0/tenants/a6beb75468d94818988bdd97335f3c1b/users/38528bc3020c4d9db0bdfa63247f744b/roles/OS-KSADM/324aa65afb4e4249a0522a8e117b604e -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 7fcb3d62905347ad8160d244c06fdd19"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.204.112.58
DEBUG:requests.packages.urllib3.connectionpool:"PUT /v2.0/tenants/a6beb75468d94818988bdd97335f3c1b/users/38528bc3020c4d9db0bdfa63247f744b/roles/OS-KSADM/324aa65afb4e4249a0522a8e117b604e HTTP/1.1" 500 3416
DEBUG:keystoneclient.session:RESP: [500] CaseInsensitiveDict({'date': 'Mon, 07 Apr 2014 21:16:59 GMT', 'content-length': '3416', 'content-type': 'text/plain', 'connection': 'close'})
RESP BODY: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 299, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 299, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 299, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 299, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 318, in __call__
    for line in req.body_file:
TypeError: 'LimitedLengthFile' object is not iterable


DEBUG:keystoneclient.session:Request returned failure status: 500
Internal Server Error (HTTP 500)