Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Depending on your version it could be due an oslo messaging bug. There are race conditions where a message is put on the queue but nova receives a message before it is done, or is waiting for the wrong message. The way to fix this was to upgrade RabbitMQ, or restart the RabbitMQ cluster and then all OpenStack services that use the messaging queue.

Of course you can also troubleshoot RabbitMQ first to verify that there aren't other issues:

See what the status is on the current node:

rabbitmqctl status

Check on the status of the cluster, see if it is partitioned or if any nodes are not part of the cluster that should be:

rabbitmqctl cluster_status

Show connections to Rabbit, see if any are being Blocked. A good state would show all Running:

rabbitmqctl list_connections

See if there are growing queues, don't show queues with 0:

rabbitmqctl list_queues |grep -v "0$"

Then you could search for alarms in the RabbitMQ log:

grep -i alarm /var/log/rabbitmq/rabbitmq-{nodenumber}.log