Ask Your Question
1

How to fix "nova.servicegroup.drivers.db MessagingTimeout: Timed out waiting for a reply to message" on a HA setup

asked 2014-12-09 13:44:58 -0500

jpmethot gravatar image

Hi,

We have a HA juno openstack cluster that we're trying to setup. Nova, Glance and Neutron are set, which is pretty close to the bare minimum to launch a VM. However, when I try to launch the VM, I get the following error message :

ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-377cb6fb-8bef-466c-8c6a-43ea7abd23cc)

On the controller in /var/log/nova/nova-api, I have :

 2014-12-09 14:31:09.260 2330 ERROR nova.api.openstack [req-66ce547a-fce5-4759-8b3d-ca19f2aac02e None] Caught error: Timed out waiting for a reply to message ID 976cb66539be48a0a0f4dd06d96d858f
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack Traceback (most recent call last):
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     return req.get_response(self.application)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     return resp(environ, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     return self._app(env, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     return resp(environ, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     return resp(environ, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     return resp(environ, start_response)
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-12-09 14:31:09.260 2330 TRACE nova.api.openstack     resp = self.call_func(req, args, *self.kwargs)
2014-12-09 14:31:09.260 2330 TRACE nova ...
(more)
edit retag flag offensive close merge delete

Comments

Which version of Oslo messaging are you using? This could be related to an issue with an older version, the fix was just to update Oslo to the latest package and restart services + RabbitMQ.

mpetason gravatar imagempetason ( 2014-12-09 14:03:12 -0500 )edit

The package I have is : Source: oslo.messaging Version: 1.4.1-0ubuntu1~cloud0

There's no more update through apt-get.

jpmethot gravatar imagejpmethot ( 2014-12-09 14:38:57 -0500 )edit

I would see if you are running into any possible bugs:

https://bugs.launchpad.net/oslo

I just know in our environment we end up updating to 1.3.1 or so, not sure if this bug is in 1.4.

mpetason gravatar imagempetason ( 2014-12-09 14:49:34 -0500 )edit

No bug report seems to cover this issue right now though. I'm hesitating making a bug report because it could actually be a config error. I'm going to continue doing further testing, potentially redo some configurations from scratch, and see what the results are.

jpmethot gravatar imagejpmethot ( 2014-12-10 07:19:35 -0500 )edit

I can confirm that this has nothing to do with the messaging server, as the problem can be reproduced on qpid. It's definitely either an openstack service misconfiguration or a bug with oslo.

jpmethot gravatar imagejpmethot ( 2014-12-10 10:01:39 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-12-10 13:09:44 -0500

jpmethot gravatar image

After searching around for about 2 days total, I ended up uninstalling nova and flushing it's database. I then reinstalled the packages and repopulated the database. I am now able to create VMs without any issue, if I don't specify a network interface, so I guess this solves the main problem.

However, I do end up with a broken neutron now (even after reinstalling packages that where removed when uninstalling nova). It's entirely possible that neutron was the root cause of this whole problem, but I have no way to prove it. I am confident I will be able to solve this neutron issue and this is outside the scope of this question anyway. So I guess this question may be considered as answered for now.

tldr;If you get a similar issue, one possible solution is to reinstall nova from scratch (and keep your config if you want).

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-12-09 13:44:58 -0500

Seen: 3,383 times

Last updated: Dec 10 '14