nova list on one tenant does not work.
When I try nova list on one specific tenant receive below error. Nothing shows up in nova.logs , other tenants work fine and other commands work fine on all tenants including this one. Have no idea where to start.
Additional Information: I have 2 controllers that are clustered and haproxy in front of controllers. Found error on the other controllerr in nova-api.logs. Listed error below.
Using Icehouse and ubuntu 14.04
nova --debug list :
New session created for: (http://10.1.0.2:8774)
INFO (connectionpool:258) Starting new HTTP connection (1): 10.1.0.2
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET /v2/3bc42bbacbd341ccaaadf718f7484fe2/servers/detail HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Tue, 18 Nov 2014 16:36:55 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-980a4a03-36e2-4212-b20c-8b05bda355f2'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
args.func(self.cs, args)
File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1138, in do_list
search_opts=search_opts)
File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 583, in list
return self._list("/servers%s%s" % (detail, query_string), "servers")
File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
_resp, body = self.api.client.get(url)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
return self._cs_request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
**kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
resp, body = self.request(url, method, **kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-980a4a03-36e2-4212-b20c-8b05bda355f2)
Nova-api.log:
2014-11-18 13:14:27.400 12147 ERROR nova.api.openstack [req-d9bb8b91-7405-4618-8b15-0db91b8afe7b f07d2a17b4324da6a6b13d470bcc84dd 3bc42bbacbd341ccaaadf718f7484fe2] Caught error: <html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack Traceback (most recent call last):
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack return req.get_response(self.application)
2014-11-18 13:14:27.400 12147 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in ...
have you checked
nova-api.log, please paste the content of nova-api.log file.Added info above. Found error on second controller.
try running this command
Nova-manage db syncfor database tables being created.that works fine, does not help with nova list. I also have HAproxy in front of the clustered controllers, wondering if something got messed up.