Ask Your Question
0

Why do I get rabbitmq broken pipe errors in my nova-api logs

asked 2015-04-23 03:10:15 -0500

SeanM gravatar image

I note that in my nova-api logs, I get frequent broken pipe ERROR's when communicating with rabbitmq. The problem seems to be associated with a heartbeat mechansim and seems to occur about every couple of hours.

<179>Apr 23 06:51:55 node-1 nova-api 2015-04-23 08:51:55.203 3598 ERROR oslo.messaging._drivers.impl_rabbit [-] [Errno 32] Broken pipe
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 849, in connection_maintenance
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     conn.send_heartbeat()
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 585, in send_heartbeat
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check()
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 268, in heartbeat_check
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 131, in heartbeat_check
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 903, in heartbeat_tick
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 880, in send_heartbeat
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     self.transport.write_frame(8, 0, bytes())
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     frame_type, channel, size, payload, 0xce,
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit     total_sent += fd.send(data[total_sent:], flags)
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-04-23 08:51:55.203 3598 TRACE oslo.messaging._drivers.impl_rabbit

The system appears to be generally working quite fine so the error does not appear to be causing any specific problems; however, we want to run a clean ship here ;-). It is an Icehouse install ... (more)

edit retag flag offensive close merge delete

Comments

More digging on this shows that the rabbitmq connections are closed due to heartbeat timeouts. The hearbeat timeouts for the rabbitmq process are configured to a default 580 seconds. However, it is not clear that the connection negotiation results in this timeout for the connection.

SeanM gravatar imageSeanM ( 2015-04-24 03:12:40 -0500 )edit

And now I see that at a certain point of time 16 connections are closed (all at the same time) and then a few minutes later the 16 connections are opened again; then some time later (usually 1-2 hours) 16 connections are closed and a few mins later 16 connections are created again.

SeanM gravatar imageSeanM ( 2015-04-24 03:34:04 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-04-30 04:56:59 -0500

SeanM gravatar image

Found the explanation for the error here:

https://bugs.launchpad.net/mos/+bug/1...

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-04-23 03:10:15 -0500

Seen: 1,553 times

Last updated: Apr 30 '15