Openstack JUNO on RHEL 7 Instance creation hang with HeartbeatTimeout and nova-api: Exception RuntimeError

asked 2015-02-01 01:28:10 -0600

HI All,

Last week I have deployed Openstack Juno on RHEL 7 using Packstack with One Controller and Compute node. Installation was successful. And I created around 30 instances and suddenly it hangs on further instance creation. Following are the errors. This seems to be happening when trying to create bulk of large (~ 150 GB) instances, once this happened even small instances can not be launched until Restart the controller and compute node.

/var/log/nova/nova-compute.log On Compute node

/var/log/nova/nova-scheduler.log , /var/log/nova/nova-conductor.log On Controller node

[-] Failed to publish message to topic 'conductor': heartbeat timeout 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid Traceback (most recent call last): 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 581, in ensure 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid return method() 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 653, in _publisher_send 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid publisher = cls(self.conf, self.session, topic=topic, **kwargs) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 409, in __init__ 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid super(TopicPublisher, self).__init__(conf, session, node_name) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 341, in __init__ 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid self.reconnect(session) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 345, in reconnect 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid self.sender = session.sender(self.address) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "<string>", line 6, in sender 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 606, in sender 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid sender._ewait(lambda: sender.linked) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 814, in _ewait 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid result = self.session._ewait(lambda: self.error or predicate(), timeout) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 580, in _ewait 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid result = self.connection._ewait(lambda: self.error or predicate(), timeout) 2015-01-31 15:51:16.803 5758 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging ... (more)

edit retag flag offensive close merge delete