Ask Your Question
2

Unable to create instances icehouse due to rabbit mq timeout

asked 2014-09-12 14:03:56 -0600

glenbot gravatar image

updated 2014-09-12 15:16:21 -0600

mpetason gravatar image

I am trying to create an instance using an Ubunut 14.04 cloud image but I am getting the following errors in nova-compute.log on the compute node.

2014-09-12 13:44:28.778 23779 DEBUG nova.compute.manager [-] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4823
2014-09-12 13:44:28.778 23779 DEBUG nova.compute.manager [-] Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4827
2014-09-12 13:44:28.803 23779 DEBUG nova.compute.manager [-] [instance: 7f30b00f-27d4-4189-b682-feece60f4f14] Skipping network cache update for instance because it is being deleted. _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4841
2014-09-12 13:44:28.804 23779 DEBUG nova.compute.manager [-] [instance: a2708a17-f8ea-4cab-9a88-7173e8b2ce74] Skipping network cache update for instance because it is Building. _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4837
2014-09-12 13:44:28.805 23779 DEBUG nova.objects.instance [-] Lazy-loading `system_metadata' on Instance uuid ac1ea94c-f1af-4767-abb8-e72015454bfe obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:519
2014-09-12 13:45:28.847 23779 ERROR nova.compute.manager [-] [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] An error occurred while refreshing the network cache.
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] Traceback (most recent call last):
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4882, in _heal_instance_info_cache
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     self._get_instance_nw_info(context, instance, use_slave=True)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1136, in _get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     instance)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 95, in wrapped
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     return func(self, context, *args, **kwargs)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 390, in get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     result = self._get_instance_nw_info(context, instance)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 406, in _get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 222, in get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     host=host, project_id=project_id)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-12 13:45:28 ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
1

answered 2014-09-12 15:21:28 -0600

mpetason gravatar image

Depending on your version it could be due an oslo messaging bug. There are race conditions where a message is put on the queue but nova receives a message before it is done, or is waiting for the wrong message. The way to fix this was to upgrade RabbitMQ, or restart the RabbitMQ cluster and then all OpenStack services that use the messaging queue.

Of course you can also troubleshoot RabbitMQ first to verify that there aren't other issues:

See what the status is on the current node:

rabbitmqctl status

Check on the status of the cluster, see if it is partitioned or if any nodes are not part of the cluster that should be:

rabbitmqctl cluster_status

Show connections to Rabbit, see if any are being Blocked. A good state would show all Running:

rabbitmqctl list_connections

See if there are growing queues, don't show queues with 0:

rabbitmqctl list_queues |grep -v "0$"

Then you could search for alarms in the RabbitMQ log:

grep -i alarm /var/log/rabbitmq/rabbitmq-{nodenumber}.log
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-09-12 14:03:56 -0600

Seen: 2,933 times

Last updated: Sep 12 '14