Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

I'm still facing that issue with timed out.

2014-10-29 17:55:47.038 4304 TRACE nova.scheduler.driver
2014-10-29 17:55:47.043 4304 WARNING nova.scheduler.driver [req-4c4498dd-6c87-4a0f-983e-afccb878d82f None] [instance: 3250c529-c18d-42c5-ad5d-7781c2cdd983] Setting instance to ERROR state.
2014-10-29 18:00:09.835 5592 INFO oslo.messaging._drivers.impl_rabbit [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 ] Connecting to AMQP server on controller:5672
2014-10-29 18:00:09.859 5592 INFO oslo.messaging._drivers.impl_rabbit [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 ] Connected to AMQP server on controller:5672
2014-10-29 18:01:09.877 5592 ERROR nova.scheduler.driver [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 None] Exception during scheduler.run_instance
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID 8a65ecd4e1c54c3186ece5f87dc920f0
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver
2014-10-29 18:01:09.881 5592 WARNING nova.scheduler.driver [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 None] [instance: c002f5a3-3a42-4f66-a045-9c95c1dda6dd] Setting instance to ERROR state.

I'm still facing that issue with timed out.

2014-10-29 17:55:47.038 4304 TRACE nova.scheduler.driver
2014-10-29 17:55:47.043 4304 WARNING nova.scheduler.driver [req-4c4498dd-6c87-4a0f-983e-afccb878d82f None] [instance: 3250c529-c18d-42c5-ad5d-7781c2cdd983] Setting instance to ERROR state.
2014-10-29 18:00:09.835 5592 18:04:11.813 5453 INFO oslo.messaging._drivers.impl_rabbit [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connecting to AMQP server on controller:5672
2014-10-29 18:00:09.859 5592 18:04:11.832 5453 INFO oslo.messaging._drivers.impl_rabbit [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connected to AMQP server on controller:5672
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 ERROR nova.scheduler.driver [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] Exception during scheduler.run_instance
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:01:09.877 5592 18:05:11.854 5453 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID 8a65ecd4e1c54c3186ece5f87dc920f0
2014-10-29 18:01:09.877 5592 TRACE nova.scheduler.driver
2014-10-29 18:01:09.881 5592 0388f5f5060241499fa4ffd53d395b6a
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver 
2014-10-29 18:05:11.858 5453 WARNING nova.scheduler.driver [req-f1f00d18-f795-4bf0-9bbc-b67c24589b37 [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] [instance: c002f5a3-3a42-4f66-a045-9c95c1dda6dd] 008e28eb-f625-4f2f-8c74-57b4d8ca0f1d] Setting instance to ERROR state.

I'm still facing that issue with timed out.out. not able to boot it not from horizon or nova.

2014-10-29 18:04:11.813 5453 INFO oslo.messaging._drivers.impl_rabbit [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connecting to AMQP server on controller:5672
2014-10-29 18:04:11.832 5453 INFO oslo.messaging._drivers.impl_rabbit [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connected to AMQP server on controller:5672
2014-10-29 18:05:11.854 5453 ERROR nova.scheduler.driver [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] Exception during scheduler.run_instance
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID 0388f5f5060241499fa4ffd53d395b6a
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver 
2014-10-29 18:05:11.858 5453 WARNING nova.scheduler.driver [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] [instance: 008e28eb-f625-4f2f-8c74-57b4d8ca0f1d] Setting instance to ERROR state.

I'm still facing that issue with timed out. not able to boot it not from horizon or nova.

2014-10-29 18:04:11.813 5453 INFO oslo.messaging._drivers.impl_rabbit [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connecting to AMQP server on controller:5672
2014-10-29 18:04:11.832 5453 INFO oslo.messaging._drivers.impl_rabbit [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connected to AMQP server on controller:5672
2014-10-29 18:05:11.854 5453 ERROR nova.scheduler.driver [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] Exception during scheduler.run_instance
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID 0388f5f5060241499fa4ffd53d395b6a
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver 
2014-10-29 18:05:11.858 5453 WARNING nova.scheduler.driver [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] [instance: 008e28eb-f625-4f2f-8c74-57b4d8ca0f1d] Setting instance to ERROR state.

nova conductor log -

2014-10-29 18:10:23.079 5084 ERROR nova.scheduler.driver [req-7e066556-1b52-4a55-bbac-4c841140ba3e None] Exception during scheduler.run_instance
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID e7b3724483194d5fb360df8806ed3fda
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver 
2014-10-29 18:10:23.085 5084 WARNING nova.scheduler.driver [req-7e066556-1b52-4a55-bbac-4c841140ba3e None] [instance: 86d7395e-7fe6-4733-ab90-22b85974c440] Setting instance to ERROR state.

I'm still facing that issue with timed out. not able to boot it not from horizon or nova.

2014-10-29 18:04:11.813 5453 INFO oslo.messaging._drivers.impl_rabbit [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connecting to AMQP server on controller:5672
2014-10-29 18:04:11.832 5453 INFO oslo.messaging._drivers.impl_rabbit [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 ] Connected to AMQP server on controller:5672
2014-10-29 18:05:11.854 5453 ERROR nova.scheduler.driver [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] Exception during scheduler.run_instance
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID 0388f5f5060241499fa4ffd53d395b6a
2014-10-29 18:05:11.854 5453 TRACE nova.scheduler.driver 
2014-10-29 18:05:11.858 5453 WARNING nova.scheduler.driver [req-aef12a6a-1b8c-4c56-8a21-d306c6b24613 None] [instance: 008e28eb-f625-4f2f-8c74-57b4d8ca0f1d] Setting instance to ERROR state.

nova conductor log -

2014-10-29 18:10:23.079 5084 ERROR nova.scheduler.driver [req-7e066556-1b52-4a55-bbac-4c841140ba3e None] Exception during scheduler.run_instance
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver Traceback (most recent call last):
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     request_spec, filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     return getattr(self.instance, __name)(*args, **kwargs)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     context, request_spec, filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     request_spec=request_spec, filter_properties=filter_properties)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     retry=self.retry)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     timeout=timeout, retry=retry)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     retry=retry)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     result = self._waiter.wait(msg_id, timeout)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     reply, ending = self._poll_connection(msg_id, timeout)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver     % msg_id)
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID e7b3724483194d5fb360df8806ed3fda
2014-10-29 18:10:23.079 5084 TRACE nova.scheduler.driver 
2014-10-29 18:10:23.085 5084 WARNING nova.scheduler.driver [req-7e066556-1b52-4a55-bbac-4c841140ba3e None] [instance: 86d7395e-7fe6-4733-ab90-22b85974c440] Setting instance to ERROR state.

rabbitmq status -

/etc/init.d# service rabbitmq-server status
Status of node rabbit@icehouse ...
[{pid,4083},
 {running_applications,[{rabbit,"RabbitMQ","3.2.4"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:32:32] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,338936808},
          {connection_procs,4343728},
          {queue_procs,1318216},
          {plugins,0},
          {other_proc,14824512},
          {mnesia,242408},
          {mgmt_db,0},
          {msg_index,80424},
          {other_ets,947576},
          {binary,291622096},
          {code,16514185},
          {atom,594537},
          {other_system,8449126}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,54062938521},
 {disk_free_limit,50000000},
 {disk_free,139060768768},
 {file_descriptors,[{total_limit,924},
                    {total_used,114},
                    {sockets_limit,829},
                    {sockets_used,112}]},
 {processes,[{limit,1048576},{used,1238}]},
 {run_queue,0},
 {uptime,5539}]
...done.