Unable to evacuate instances

asked 2016-03-30 05:27:26 -0600

zekken gravatar image

updated 2016-03-30 22:57:07 -0600

Bipin gravatar image

Whenever I am trying to evacuate more than one instance to other compute nodes, the first one gets evacuated and rebuilt, but the other instances gets evacuated (nova-evacuate command runs without any issues), but does not rebuilds, gets stuck in rebuilding state and I get the following error in the dashboard : Error: Failed to launch instance "test-4f4793d5-5862-42c2-8d22-e13e6e139af4": Please try again later [Error: Timed out waiting for a reply to message ID . My nova-conductor.log is :

2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 670, in ensure
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     return method()
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 785, in _publish
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 375, in __init__
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 339, in __init__
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 347, in reconnect
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 620, in exchange_declare
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
2016-03-29 18:34:09.479 1265 TRACE oslo.messaging._drivers.impl_rabbit   File ...
(more)
edit retag flag offensive close merge delete

Comments

It seems to me as a rabbitmq-timeout issue. I have tried increasing the descriptor limits but it doesn't impact anything.

zekken gravatar imagezekken ( 2016-03-30 07:27:41 -0600 )edit