Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

glance HTTPInternalServerError (HTTP 500)

Hi,

I red many questions on HTTPInternalServerError in forum, while some of them are very close to my condition, I still didn't find the way to fix it.

The registry.log was fine and nothing abnormal reported.

Here is what I get from "$ glance image-list": HTTPInternalServerError (HTTP 500)

Here is the traceback in api.log:

2015-01-12 13:28:43.508 11354 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:44
2015-01-12 13:28:43.508 11354 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:57
2015-01-12 13:28:43.509 11354 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v1 process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:69
2015-01-12 13:28:43.509 11354 DEBUG glance.api.middleware.version_negotiation [-] new path /v1/images/detail process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:70
2015-01-12 13:28:43.510 11354 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
2015-01-12 13:28:43.510 11354 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707
2015-01-12 13:28:43.541 11354 DEBUG keystonemiddleware.auth_token [-] Server reports support for api versions: v3.0, v2.0 _get_supported_versions /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:640
2015-01-12 13:28:43.541 11354 INFO keystonemiddleware.auth_token [-] Auth Token confirmed use of v2.0 apis
2015-01-12 13:28:43.695 11354 DEBUG keystonemiddleware.auth_token [-] Storing token in cache store /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:1435
2015-01-12 13:28:43.695 11354 DEBUG keystonemiddleware.auth_token [-] Received request from user: d26d834886504c09896217c2abec1097 with project_id : e7225a5056e540999c5811065f758800 and roles: admin  _build_user_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:971
2015-01-12 13:28:43.702 11354 DEBUG routes.middleware [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Matched GET /images/detail __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2015-01-12 13:28:43.703 11354 DEBUG routes.middleware [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Route path: '/images/detail', defaults: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x7fad9f314f50>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2015-01-12 13:28:43.703 11354 DEBUG routes.middleware [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Match dict: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x7fad9f314f50>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2015-01-12 13:28:43.705 11354 DEBUG glance.common.client [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Constructed URL: http://0.0.0.0:9191/images/detail?sort_key=name&sort_dir=asc&limit=20 _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:398
2015-01-12 13:28:43.708 11354 INFO glance.registry.client.v1.client [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Registry client request GET /images/detail raised ServerError
2015-01-12 13:28:43.712 11354 INFO glance.wsgi.server [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
    return self._app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 683, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 707, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 347, in detail
    images = registry.get_images_detail(req.context, **params)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py", line 150, in get_images_detail
    return c.get_images_detailed(**kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 144, in get_images_detailed
    res = self.do_request("GET", "/images/detail", params=params)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 130, in do_request
    'exc_name': exc_name})
  File "/usr/lib/python2.7/dist-packages/glance/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 115, in do_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 68, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 373, in do_request
    headers=copy.deepcopy(headers))
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 85, in wrapped
    return func(self, method, url, body, headers)
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 531, in _do_request
    raise exception.ServerError()
ServerError: The request returned 500 Internal Server Error.
2015-01-12 13:28:43.712 11354 INFO glance.wsgi.server [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] 10.0.0.11 - - [12/Jan/2015 13:28:43] "GET /v1/images/detail?sort_key=name&sort_dir=asc&limit=20 HTTP/1.1" 500 139 0.205499

I have changed the encoding of glance db tables into 'utf-8' and I'm sure that the tenant, service and user etc are correctly setup in keystone. And the configuration of keystonemiddleware is listed as below:

[keystone_authtoken]
identity_uri = http://10.0.0.11:35357
auth_uri = http://10.0.0.11:5000/v2.0
admin_tenant_name = service
admin_user = glance
admin_password = PASSWORD
revocation_cache_time = 10

can anyone help me with this problem?

glance HTTPInternalServerError (HTTP 500)

Hi,

I red many questions on HTTPInternalServerError in forum, while some of them are very close to my condition, I still didn't find the way to fix it.

The registry.log was fine and nothing abnormal reported.

Here is what I get from "$ glance image-list": HTTPInternalServerError (HTTP 500)

Here is the traceback in api.log:

2015-01-12 13:28:43.508 11354 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:44
2015-01-12 13:28:43.508 11354 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:57
2015-01-12 13:28:43.509 11354 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v1 process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:69
2015-01-12 13:28:43.509 11354 DEBUG glance.api.middleware.version_negotiation [-] new path /v1/images/detail process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:70
2015-01-12 13:28:43.510 11354 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
2015-01-12 13:28:43.510 11354 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707
2015-01-12 13:28:43.541 11354 DEBUG keystonemiddleware.auth_token [-] Server reports support for api versions: v3.0, v2.0 _get_supported_versions /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:640
2015-01-12 13:28:43.541 11354 INFO keystonemiddleware.auth_token [-] Auth Token confirmed use of v2.0 apis
2015-01-12 13:28:43.695 11354 DEBUG keystonemiddleware.auth_token [-] Storing token in cache store /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:1435
2015-01-12 13:28:43.695 11354 DEBUG keystonemiddleware.auth_token [-] Received request from user: d26d834886504c09896217c2abec1097 with project_id : e7225a5056e540999c5811065f758800 and roles: admin  _build_user_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:971
2015-01-12 13:28:43.702 11354 DEBUG routes.middleware [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Matched GET /images/detail __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2015-01-12 13:28:43.703 11354 DEBUG routes.middleware [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Route path: '/images/detail', defaults: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x7fad9f314f50>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2015-01-12 13:28:43.703 11354 DEBUG routes.middleware [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Match dict: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x7fad9f314f50>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2015-01-12 13:28:43.705 11354 DEBUG glance.common.client [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Constructed URL: http://0.0.0.0:9191/images/detail?sort_key=name&sort_dir=asc&limit=20 _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:398
2015-01-12 13:28:43.708 11354 INFO glance.registry.client.v1.client [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Registry client request GET /images/detail raised ServerError
2015-01-12 13:28:43.712 11354 INFO glance.wsgi.server [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
    return self._app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 683, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 707, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 347, in detail
    images = registry.get_images_detail(req.context, **params)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py", line 150, in get_images_detail
    return c.get_images_detailed(**kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 144, in get_images_detailed
    res = self.do_request("GET", "/images/detail", params=params)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 130, in do_request
    'exc_name': exc_name})
  File "/usr/lib/python2.7/dist-packages/glance/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py", line 115, in do_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 68, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 373, in do_request
    headers=copy.deepcopy(headers))
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 85, in wrapped
    return func(self, method, url, body, headers)
  File "/usr/lib/python2.7/dist-packages/glance/common/client.py", line 531, in _do_request
    raise exception.ServerError()
ServerError: The request returned 500 Internal Server Error.
2015-01-12 13:28:43.712 11354 INFO glance.wsgi.server [b32aa3da-9fce-44bb-89a8-93623d090ba5 d26d834886504c09896217c2abec1097 e7225a5056e540999c5811065f758800 - - -] 10.0.0.11 - - [12/Jan/2015 13:28:43] "GET /v1/images/detail?sort_key=name&sort_dir=asc&limit=20 HTTP/1.1" 500 139 0.205499

I have changed the encoding of glance db tables into 'utf-8' and I'm sure that the tenant, service and user etc are correctly setup in keystone. And the configuration of keystonemiddleware is listed as below:

[keystone_authtoken]
identity_uri = http://10.0.0.11:35357
auth_uri = http://10.0.0.11:5000/v2.0
admin_tenant_name = service
admin_user = glance
admin_password = PASSWORD
revocation_cache_time = 10

can anyone help me with this problem?problem? If you need more details, just let me know.