Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Unable to create instances icehouse due to rabbit mq timeout

I am trying to create an instance using an Ubunut 14.04 cloud image but I am getting the following errors in nova-compute.log on the compute node.

2014-09-12 13:44:28.778 23779 DEBUG nova.compute.manager [-] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4823 2014-09-12 13:44:28.778 23779 DEBUG nova.compute.manager [-] Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4827 2014-09-12 13:44:28.803 23779 DEBUG nova.compute.manager [-] [instance: 7f30b00f-27d4-4189-b682-feece60f4f14] Skipping network cache update for instance because it is being deleted. _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4841 2014-09-12 13:44:28.804 23779 DEBUG nova.compute.manager [-] [instance: a2708a17-f8ea-4cab-9a88-7173e8b2ce74] Skipping network cache update for instance because it is Building. _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4837 2014-09-12 13:44:28.805 23779 DEBUG nova.objects.instance [-] Lazy-loading `system_metadata' on Instance uuid ac1ea94c-f1af-4767-abb8-e72015454bfe obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:519 2014-09-12 13:45:28.847 23779 ERROR nova.compute.manager [-] [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] An error occurred while refreshing the network cache. 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] Traceback (most recent call last): 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4882, in _heal_instance_info_cache 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] self._get_instance_nw_info(context, instance, use_slave=True) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1136, in _get_instance_nw_info 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] instance) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 95, in wrapped 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] return func(self, context, args, *kwargs) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 390, in get_instance_nw_info 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] result = self._get_instance_nw_info(context, instance) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 406, in _get_instance_nw_info 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] nw_info = self.network_rpcapi.get_instance_nw_info(context, **args) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 222, in get_instance_nw_info 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] host=host, project_id=project_id) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] wait_for_reply=True, timeout=timeout) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] timeout=timeout) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] return self._send(target, ctxt, message, wait_for_reply, timeout) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 403, in _send 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] result = self._waiter.wait(msg_id, timeout) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 280, in wait 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] reply, ending, trylock = self._poll_queue(msg_id, timeout) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 220, in _poll_queue 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] message = self.waiters.get(msg_id, timeout) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 126, in get 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] 'to message ID %s' % msg_id) 2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] MessagingTimeout: Timed out waiting for a reply to message ID f79fcd85bc774700986ead7f9cf6aef2

The connection to rabbitmq seems to be working when nova-compute is restarted. I have seen this issue in many different threads with no resolution. Anyone have an idea on what could be causing this?

click to hide/show revision 2
No.2 Revision

Unable to create instances icehouse due to rabbit mq timeout

I am trying to create an instance using an Ubunut 14.04 cloud image but I am getting the following errors in nova-compute.log on the compute node.

2014-09-12 13:44:28.778 23779 DEBUG nova.compute.manager [-] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4823
2014-09-12 13:44:28.778 23779 DEBUG nova.compute.manager [-] Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4827
2014-09-12 13:44:28.803 23779 DEBUG nova.compute.manager [-] [instance: 7f30b00f-27d4-4189-b682-feece60f4f14] Skipping network cache update for instance because it is being deleted. _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4841
2014-09-12 13:44:28.804 23779 DEBUG nova.compute.manager [-] [instance: a2708a17-f8ea-4cab-9a88-7173e8b2ce74] Skipping network cache update for instance because it is Building. _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4837
2014-09-12 13:44:28.805 23779 DEBUG nova.objects.instance [-] Lazy-loading `system_metadata' on Instance uuid ac1ea94c-f1af-4767-abb8-e72015454bfe obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:519
2014-09-12 13:45:28.847 23779 ERROR nova.compute.manager [-] [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] An error occurred while refreshing the network cache.
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] Traceback (most recent call last):
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4882, in _heal_instance_info_cache
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     self._get_instance_nw_info(context, instance, use_slave=True)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1136, in _get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     instance)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 95, in wrapped
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     return func(self, context, args, *kwargs)
*args, **kwargs)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 390, in get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     result = self._get_instance_nw_info(context, instance)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 406, in _get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 222, in get_instance_nw_info
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     host=host, project_id=project_id)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     wait_for_reply=True, timeout=timeout)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     timeout=timeout)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 403, in _send
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     result = self._waiter.wait(msg_id, timeout)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 280, in wait
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     reply, ending, trylock = self._poll_queue(msg_id, timeout)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 220, in _poll_queue
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     message = self.waiters.get(msg_id, timeout)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 126, in get
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe]     'to message ID %s' % msg_id)
2014-09-12 13:45:28.847 23779 TRACE nova.compute.manager [instance: ac1ea94c-f1af-4767-abb8-e72015454bfe] MessagingTimeout: Timed out waiting for a reply to message ID f79fcd85bc774700986ead7f9cf6aef2

f79fcd85bc774700986ead7f9cf6aef2

The connection to rabbitmq seems to be working when nova-compute is restarted. I have seen this issue in many different threads with no resolution. Anyone have an idea on what could be causing this?