Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation.

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the controller node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     project_id=project_id)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1236, in _shutdown_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 681, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 360, in get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 368, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 787, in _build_network_info_model
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp
2013-12-19 08:47:15.746 1507 ERROR nova.network.quantumv2 [-] _get_auth_token() failed
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Traceback (most recent call last):
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/__init__.py", line 40, in _get_auth_token
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     httpclient.authenticate()
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/quantumclient/client.py", line 198, in authenticate
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     raise exceptions.Unauthorized(message=body)
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Unauthorized: Request Timeout

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation.

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the controller compute node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     project_id=project_id)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1236, in _shutdown_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 681, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 360, in get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 368, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 787, in _build_network_info_model
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp
2013-12-19 08:47:15.746 1507 ERROR nova.network.quantumv2 [-] _get_auth_token() failed
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Traceback (most recent call last):
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/__init__.py", line 40, in _get_auth_token
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     httpclient.authenticate()
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/quantumclient/client.py", line 198, in authenticate
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     raise exceptions.Unauthorized(message=body)
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Unauthorized: Request Timeout

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation.

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the compute node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     project_id=project_id)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1236, in _shutdown_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 681, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 360, in get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 368, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 787, in _build_network_info_model
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp
2013-12-19 08:47:15.746 1507 ERROR nova.network.quantumv2 [-] _get_auth_token() failed
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Traceback (most recent call last):
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/__init__.py", line 40, in _get_auth_token
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     httpclient.authenticate()
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/quantumclient/client.py", line 198, in authenticate
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     raise exceptions.Unauthorized(message=body)
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Unauthorized: Request Timeout

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation.operation. How do I generate tokens?

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the compute node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     project_id=project_id)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1236, in _shutdown_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 681, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 360, in get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 368, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 787, in _build_network_info_model
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp
2013-12-19 08:47:15.746 1507 ERROR nova.network.quantumv2 [-] _get_auth_token() failed
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Traceback (most recent call last):
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/__init__.py", line 40, in _get_auth_token
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     httpclient.authenticate()
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/quantumclient/client.py", line 198, in authenticate
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     raise exceptions.Unauthorized(message=body)
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Unauthorized: Request Timeout

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation. How do I generate tokens?

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the compute node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     project_id=project_id)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1236, in _shutdown_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 681, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 360, in get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 368, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 787, in _build_network_info_model
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp
2013-12-19 08:47:15.746 1507 ERROR nova.network.quantumv2 [-] _get_auth_token() failed
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Traceback (most recent call last):
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/__init__.py", line 40, in _get_auth_token
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     httpclient.authenticate()
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/quantumclient/client.py", line 198, in authenticate
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     raise exceptions.Unauthorized(message=body)
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Unauthorized: Request Timeout

After I restarted all services and got a new token this is error I get from controller node's "nova-api.log":

2013-12-24 08:00:53.684 ERROR nova.api.openstack [req-0d214c1a-652b-4e23-b44b-d3af8a30dc34 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Caught error: Request Timeout
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack Traceback (most recent call last):
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return req.get_response(self.application)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     application, catch_exc_info=False)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return self.app(env, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     response = self.app(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 890, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     content_type, body, accept)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 969, in _process_stack
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     request, action_args)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 863, in post_process_extensions
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     **action_args)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 537, in detail
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 487, in _extend_servers
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     .get_instances_security_groups_bindings(context))
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/quantum_driver.py", line 252, in get_instances_security_groups_bindings
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     ports = quantum.list_ports().get('ports')
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     **_params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     res = self.get(path, params=params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     headers=headers, params=params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     headers=headers, params=params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     message=message)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack QuantumClientException: Request Timeout
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack
2013-12-24 08:00:53.687 INFO nova.api.openstack [req-0d214c1a-652b-4e23-b44b-d3af8a30dc34 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] http://192.168.100.51:8774/v2/3b6d29d5916c4723ab10e99f180f3b61/servers/detail returned with HTTP 500
2013-12-24 08:00:53.689 INFO nova.osapi_compute.wsgi.server [req-0d214c1a-652b-4e23-b44b-d3af8a30dc34 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] 192.168.100.51 "GET /v2/3b6d29d5916c4723ab10e99f180f3b61/servers/detail HTTP/1.1" status: 500 len: 335 time: 78.1657879

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation. How do I generate tokens?tokens? QuantumClientException: Request Timeout

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the compute node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     project_id=project_id)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1236, in _shutdown_instance
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 681, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 360, in get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 368, in _get_instance_nw_info
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 787, in _build_network_info_model
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp
2013-12-19 08:47:15.746 1507 ERROR nova.network.quantumv2 [-] _get_auth_token() failed
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Traceback (most recent call last):
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/__init__.py", line 40, in _get_auth_token
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     httpclient.authenticate()
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2   File "/usr/lib/python2.7/dist-packages/quantumclient/client.py", line 198, in authenticate
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2     raise exceptions.Unauthorized(message=body)
2013-12-19 08:47:15.746 1507 TRACE nova.network.quantumv2 Unauthorized: Request Timeout

After I restarted all services and got a new token this is error I get from controller node's "nova-api.log":

2013-12-24 08:00:53.684 ERROR nova.api.openstack [req-0d214c1a-652b-4e23-b44b-d3af8a30dc34 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Caught error: Request Timeout
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack Traceback (most recent call last):
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return req.get_response(self.application)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     application, catch_exc_info=False)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return self.app(env, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     response = self.app(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return resp(environ, start_response)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 890, in __call__
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     content_type, body, accept)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 969, in _process_stack
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     request, action_args)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 863, in post_process_extensions
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     **action_args)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 537, in detail
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 487, in _extend_servers
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     .get_instances_security_groups_bindings(context))
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/quantum_driver.py", line 252, in get_instances_security_groups_bindings
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     ports = quantum.list_ports().get('ports')
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     **_params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     res = self.get(path, params=params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     headers=headers, params=params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     headers=headers, params=params)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack     message=message)
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack QuantumClientException: Request Timeout
2013-12-24 08:00:53.684 1607 TRACE nova.api.openstack
2013-12-24 08:00:53.687 INFO nova.api.openstack [req-0d214c1a-652b-4e23-b44b-d3af8a30dc34 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] http://192.168.100.51:8774/v2/3b6d29d5916c4723ab10e99f180f3b61/servers/detail returned with HTTP 500
2013-12-24 08:00:53.689 INFO nova.osapi_compute.wsgi.server [req-0d214c1a-652b-4e23-b44b-d3af8a30dc34 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] 192.168.100.51 "GET /v2/3b6d29d5916c4723ab10e99f180f3b61/servers/detail HTTP/1.1" status: 500 len: 335 time: 78.1657879