Revision history [back]

click to hide/show revision 1
initial version

Why do I get rabbitmq broken pipe errors in my nova-api logs

I note that in my nova-api logs, I get frequent broken pipe ERROR's when communicating with rabbitmq. The problem seems to be associated with a heartbeat mechansim and seems to occur about every couple of hours.

<179>Apr 23 06:51:55 node-1 nova-api 2015-04-23 08:51:55.203 3598 ERROR oslo.messaging._drivers.impl_rabbit [-] [Errno 32] Broken pipe
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 849, in connection_maintenance
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     conn.send_heartbeat()
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 585, in send_heartbeat
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check()
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 268, in heartbeat_check
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 131, in heartbeat_check
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 903, in heartbeat_tick
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 880, in send_heartbeat
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     self.transport.write_frame(8, 0, bytes())
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     frame_type, channel, size, payload, 0xce,
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     total_sent += fd.send(data[total_sent:], flags)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit

The system appears to be generally working quite fine so the error does not appear to be causing any specific problems; however, we want to run a clean ship here ;-). It is an Icehouse install, deployed with Mirantis Fuel.

I'm not really sure where to look to find the root of this problem - any pointers?

Thanks, Seán.