Ask Your Question

Debugging QPID messaging.

asked 2014-02-13 07:06:06 -0600

Krist gravatar image

updated 2014-02-13 14:09:32 -0600

koolhead17 gravatar image


We are having some problems with cinder and want to properly find out what causes them. In order to do this I think need to see more of what happens in QPID.

Basically I see cinder-scheduler send messages to a cinder-volume node. I have debugging enabled on both nodes, so I expect to see in the log of the cinder-volume node each message that cinder-scheduler sends it. I am however not seeing anything. I see in the debug log of cinder-scheduler that it sends messages. However I do not see in the debug log in cinder-volume that it is receiving those messages. Is there a way to observer the messages as they pass through qpid? It looks as if something is consuming those messages, but it is not the process that is supposed to consume them. I want to find this out. So I am looking at something that just gives me a good log of everything that passes through qpid. I want to see the messages, who send them, who picked them up, what the content and methods were etc.. We are running qpid as part of Redhat Openstack 4 How do I do this?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2014-06-13 04:24:39 -0600

Gordon Sim gravatar image

The first step is perhaps to use qpid-stat to get some information about the message flows at a high level. Running qpid-stat -c will list all connections, and you can see the process name and host for these. Running qpid-stat -e will list all exchanges. Of interest here are any exchanges for which there are no binding but also showing lots of dropped messages as this could indicate a communication failure of sort. Running qpid-stat -q will ist all queues, of particular interest is the number of message currently enqueued (a large backlog indicates something going wrong, and the enqueue/dequeue count. Also interesting is the number of consumers on the queue; a queue with no consumers indicates a possible problem, especially if it has messages in it. Finally qpid-stat -u lists all subscriptions.

Using this information you can usually piece together a better picture of what is happening. Running the commands more than once lets you observe if there are changes to the various message counts (i.e. whether messages have been flowing through those channels).

If you really need it, you can turn on protocol tracing for the broker (e.g. specify --log-enable notify+ --log-enable trace+:Protocol) or better still use wireshark. However be aware that this is very detailed, protocol level information. Unfortunately there is no per-message tracing function.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2014-02-13 07:06:06 -0600

Seen: 1,099 times

Last updated: Jun 13 '14