Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

asked 2014-11-18 10:39:07 -0600

bgyako gravatar image

nova list on one tenant does not work.

When I try nova list on one specific tenant receive below error. Nothing shows up in nova.logs , other tenants work fine and other commands work fine on all tenants including this one. Have no idea where to start.

Using Icehouse and ubuntu 14.04

nova --debug list :

New session created for: (http://10.1.0.2:8774)
INFO (connectionpool:258) Starting new HTTP connection (1): 10.1.0.2
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Tue, 18 Nov 2014 16:36:55 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-980a4a03-36e2-4212-b20c-8b05bda355f2'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1138, in do_list
    search_opts=search_opts)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 583, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)

nova list on one tenant does not work.

When I try nova list on one specific tenant receive below error. Nothing shows up in nova.logs , other tenants work fine and other commands work fine on all tenants including this one. Have no idea where to start.

Additional Information: I have 2 controllers that are clustered. Found error on the other controllerr in nova-api.logs. Listed error below.

Using Icehouse and ubuntu 14.04

nova --debug list :

New session created for: (http://10.1.0.2:8774)
INFO (connectionpool:258) Starting new HTTP connection (1): 10.1.0.2
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Tue, 18 Nov 2014 16:36:55 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-980a4a03-36e2-4212-b20c-8b05bda355f2'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1138, in do_list
    search_opts=search_opts)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 583, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)

Nova-api.log:

2014-11-18 13:14:27.400 12147 ERROR nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] Caught error: <html><body>

400 Bad request

Your browser sent an invalid request. </body></html>

2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Traceback (most recent call last): 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return req.get_response(self.application) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack application, catch_exc_info=False) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return resp(environ, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return self.app(env, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return resp(environ, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return resp(environ, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack response = self.app(environ, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return resp(environ, start_response) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack resp = self.call_func(req, args, self.kwargs) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return self.func(req, *args, *kwargs) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__ 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack content_type, body, accept) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1009, in _procenter code hereess_stack 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack request, action_args) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack *action_args) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack self._extend_servers(req, list(resp_obj.obj['servers'])) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack servers)) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack ports = self._get_ports_from_server_list(servers, neutron) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack ports.extend(neutron.list_ports(search_opts).get('ports')) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack ret = self.function(instance, *args, *kwargs) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 306, in list_ports 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack *_params) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1250, in list 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack for r in self._pagination(collection, path, *params): 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1263, in _pagination 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack res = self.get(path, params=params) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1236, in get 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack headers=headers, params=params) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1221, in retry_request 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack headers=headers, params=params) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1164, in do_request 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack message=message) 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack NeutronClientException: <html><body>

400 Bad request

2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Your browser sent an invalid request. 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack </body></html> 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack 2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack 2014-11-18 13:14:27.403 12147 INFO nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] http://10.1.0.2:8774/v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail returned with HTTP 500 2014-11-18 13:14:27.406 12147 INFO nova.osapi_compute.wsgi.server [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] 10.1.0.3 "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" status: 500 len: 335 time: 0.5533540 2014-11-18 13:14:28.150 12109 INFO nova.ec2.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 404 len: 196 time: 0.0017850 2014-11-18 13:14:28.153 12151 INFO nova.osapi_compute.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 200 len: 278 time: 0.0008380

nova list on one tenant does not work.

When I try nova list on one specific tenant receive below error. Nothing shows up in nova.logs , other tenants work fine and other commands work fine on all tenants including this one. Have no idea where to start.

Additional Information: I have 2 controllers that are clustered. Found error on the other controllerr in nova-api.logs. Listed error below.

Using Icehouse and ubuntu 14.04

nova --debug list :

:
 
New session created for: (http://10.1.0.2:8774)
INFO (connectionpool:258) Starting new HTTP connection (1): 10.1.0.2
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Tue, 18 Nov 2014 16:36:55 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-980a4a03-36e2-4212-b20c-8b05bda355f2'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1138, in do_list
    search_opts=search_opts)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 583, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)

 
Nova-api.log:
 
2014-11-18 13:14:27.400 12147 ERROR nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] 3bc42bbacbd341ccaaadf718f7484fe2]
Caught error: <html><body>
400 Bad request

Your request</h1>Your browser sent an invalid request.
</body></html>
 
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Traceback (most recent call last):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return req.get_response(self.application)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return self.app(env, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131,  in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     resp = self.call_func(req, args, self.kwargs)
*args, **self.kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return self.func(req, *args, *kwargs)
**kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     content_type, body, accept)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1009, in _procenter _proc`enter code hereess_stack
here`ess_stack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     request, action_args)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     *action_args)
**action_args)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     servers))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ports.extend(neutron.list_ports(search_opts).get('ports'))
ports.extend(neutron.list_ports(**search_opts).get('ports'))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ret = self.function(instance, *args, *kwargs)
**kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 306, in list_ports
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     *_params)
**_params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1250, in list
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     for r in self._pagination(collection, path, *params):
**params):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     res = self.get(path, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1236, in get
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     headers=headers, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     headers=headers, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1164, in do_request
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     message=message)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack NeutronClientException: <html><body>
400 <html><body><h1>400 Bad request
 request</h1>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Your browser sent an invalid request.
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack </body></html>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.403 12147 INFO nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] http://10.1.0.2:8774/v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail returned with HTTP 500
2014-11-18 13:14:27.406 12147 INFO nova.osapi_compute.wsgi.server [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] 10.1.0.3 "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" status: 500 len: 335 time: 0.5533540
2014-11-18 13:14:28.150 12109 INFO nova.ec2.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 404 len: 196 time: 0.0017850
2014-11-18 13:14:28.153 12151 INFO nova.osapi_compute.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 200 len: 278 time: 0.0008380
0.0008380
click to hide/show revision 4
No.4 Revision

updated 2014-11-18 12:25:06 -0600

Syed Awais Ali gravatar image

nova list on one tenant does not work.

When I try nova list on one specific tenant receive below error. Nothing shows up in nova.logs , other tenants work fine and other commands work fine on all tenants including this one. Have no idea where to start.

Additional Information: I have 2 controllers that are clustered. Found error on the other controllerr in nova-api.logs. Listed error below.

Using Icehouse and ubuntu 14.04

nova --debug list :
 
:

New session created for: (http://10.1.0.2:8774)
INFO (connectionpool:258) Starting new HTTP connection (1): 10.1.0.2
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Tue, 18 Nov 2014 16:36:55 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-980a4a03-36e2-4212-b20c-8b05bda355f2'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1138, in do_list
    search_opts=search_opts)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 583, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)

 
Nova-api.log:
 
2014-11-18 13:14:27.400 12147 ERROR nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2]
3bc42bbacbd341ccaaadf718f7484fe2] Caught error: 400 <html><body><h1>400 Bad request</h1>Your request</h1>
Your browser sent an invalid request.
</body></html>

2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Traceback (most recent call last):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return req.get_response(self.application)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return self.app(env, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131,  in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     content_type, body, accept)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1009, in _proc`enter code here`ess_stack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     request, action_args)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     **action_args)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     servers))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 306, in list_ports
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     **_params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1250, in list
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     res = self.get(path, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1236, in get
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     headers=headers, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     headers=headers, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1164, in do_request
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     message=message)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack NeutronClientException: <html><body><h1>400 Bad request</h1>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Your browser sent an invalid request.
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack </body></html>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.403 12147 INFO nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] http://10.1.0.2:8774/v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail returned with HTTP 500
2014-11-18 13:14:27.406 12147 INFO nova.osapi_compute.wsgi.server [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] 10.1.0.3 "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" status: 500 len: 335 time: 0.5533540
2014-11-18 13:14:28.150 12109 INFO nova.ec2.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 404 len: 196 time: 0.0017850
2014-11-18 13:14:28.153 12151 INFO nova.osapi_compute.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 200 len: 278 time: 0.0008380

nova list on one tenant does not work.

When I try nova list on one specific tenant receive below error. Nothing shows up in nova.logs , other tenants work fine and other commands work fine on all tenants including this one. Have no idea where to start.

Additional Information: I have 2 controllers that are clustered. clustered and haproxy in front of controllers. Found error on the other controllerr in nova-api.logs. Listed error below.

Using Icehouse and ubuntu 14.04

nova --debug list :

New session created for: (http://10.1.0.2:8774)
INFO (connectionpool:258) Starting new HTTP connection (1): 10.1.0.2
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Tue, 18 Nov 2014 16:36:55 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-980a4a03-36e2-4212-b20c-8b05bda355f2'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1138, in do_list
    search_opts=search_opts)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 583, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)

Nova-api.log:

2014-11-18 13:14:27.400 12147 ERROR nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] Caught error: <html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Traceback (most recent call last):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return req.get_response(self.application)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return self.app(env, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return resp(environ, start_response)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     content_type, body, accept)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1009, in _proc`enter code here`ess_stack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     request, action_args)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     **action_args)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     servers))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 306, in list_ports
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     **_params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1250, in list
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     res = self.get(path, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1236, in get
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     headers=headers, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     headers=headers, params=params)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1164, in do_request
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack     message=message)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack NeutronClientException: <html><body><h1>400 Bad request</h1>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Your browser sent an invalid request.
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack </body></html>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack
2014-11-18 13:14:27.403 12147 INFO nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] http://10.1.0.2:8774/v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail returned with HTTP 500
2014-11-18 13:14:27.406 12147 INFO nova.osapi_compute.wsgi.server [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] 10.1.0.3 "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" status: 500 len: 335 time: 0.5533540
2014-11-18 13:14:28.150 12109 INFO nova.ec2.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 404 len: 196 time: 0.0017850
2014-11-18 13:14:28.153 12151 INFO nova.osapi_compute.wsgi.server [-] 10.1.0.3 "OPTIONS / HTTP/1.0" status: 200 len: 278 time: 0.0008380