Ask Your Question
0

Grizzly - ERROR: The server has either erred or is incapable of performing the requested operation. How do I generate tokens? QuantumClientException: Request Timeout [closed]

asked 2013-12-19 05:37:50 -0500

amber_nic gravatar image

updated 2013-12-24 04:12:29 -0500

Hi all,

I deployed a farm following this guide. It's been three months running in an stable way with 13 tenants but a couple days ago I have almost no control over the the system. It doesn't let me delete instances and sometimes when I type "nova list" after a long while it says: "ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" I attached a selection of the compute node's "nova-compute.log" I don't understand why this is happening because I didn't make a single change in any of the configuration files. Hope someone can help me.

2013-12-19 08:46:58.427 ERROR nova.openstack.common.rpc.amqp [req-6017dcc1-21ca-4f2b-b172-c7d6887a8084 d147378610b84425a35f3ed1fdaebcd8 3b6d29d5916c4723ab10e99f180f3b61] Exception during message handling
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-12-19 08:46:58.427 1507 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova ...
(more)
edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by dheeru
close date 2014-01-03 10:54:08.598777

1 answer

Sort by ยป oldest newest most voted
0

answered 2013-12-19 07:19:14 -0500

dheeru gravatar image

updated 2014-01-03 10:54:53 -0500

In summary it is authentication/authorization issue. You are trying to access after so many days. Tokens would have expired. Ensure that you are setting the right credentials in in localrc file or environment variables. Also Check whether all the services are running.

Summary :

I did the live troubleshooting using team viewer. Tokens were expired. What we did was to regenerate the tokens. We edited the keystone.conf with token_format=UUID and restarted the keystone as well.

Post this everything started working fine.

edit flag offensive delete link more

Comments

Thank you very much! I'll follow your advice.

amber_nic gravatar imageamber_nic ( 2013-12-19 07:30:10 -0500 )edit

Hi dheeru, the services are all running. I didn't change the credentials and they work fine. However I don't know how to generate new tokens. Could you please point me to some link or give me some clue on how to do it ?

amber_nic gravatar imageamber_nic ( 2013-12-23 05:06:23 -0500 )edit

1. Restart all your services once. 2. Can you run your command with --debug option and see what is debug printed 3. Are you able to run 'keystone user-list", "keystone tenant-list" etc commands ? 4. What are your settings of OS_USERNAME, OS_TENANTNAME etc ? 5. Can you run the command 'keystone token-get' ? Hope you are able to see the new token.

dheeru gravatar imagedheeru ( 2013-12-23 08:25:20 -0500 )edit

Hi dheeru, I succesfully performed the checks in all the points. Now it doesn't seem to be an authorization problem. I attached above the new error it gives: "QuantumClientException: Request Timeout"

amber_nic gravatar imageamber_nic ( 2013-12-24 04:10:18 -0500 )edit

It is getting into issue when trying to get the details about each instance. I'm assuming that it is multi node installation. Currently tenant in question is 3b6d29d5916c4723ab10e99f180f3b61. Since you said there are 13 tenants, can verify this tenant(Run the command keystone tenant-list) ? When quantum is trying to get the list of ports it is failing. You can try running the command 'quantum port-list' and see what happens. I still suspect some communication issue or service is down.

dheeru gravatar imagedheeru ( 2013-12-25 11:50:30 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-12-19 05:37:50 -0500

Seen: 1,109 times

Last updated: Jan 03 '14