API calls routed to wrong service endpoint

asked 2014-11-13 12:20:34 -0600

dt.turner gravatar image

updated 2014-11-17 10:48:15 -0600

Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from https://github.com/stackforge/mistral (github) based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:


I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:


auth_host =

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx


Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on. Any general tips for troubleshooting incorrectly routed API calls outside of verifying the keystone service endpoint configuration?

Update: OK.. so things just got a bit weirder. I deleted the mistral service and endpoint from keystone and bounced the keystone service. I left mistral server running within it's VM and can still see that heat requests are getting routed to it via the log errors.

Update: Added the mistral keystone service and endpoint back in order to see if the problem somehow had to do with the original service / endpoint entries, but still seeing the same ... (more)

edit retag flag offensive close merge delete


It sounds like both heat-engine and Mistral are listening on the same RPC queue for some reason.

zaneb gravatar imagezaneb ( 2014-12-08 11:20:04 -0600 )edit

I would agree.. unfortunately (or fortunately, depending on how you look at it) I haven't been able to reproduce since enabling v3 Keystone API and redirecting Mistral to the v3 auth endpoint. Once I made this change, the problem ceased to occur.

dt.turner gravatar imagedt.turner ( 2014-12-08 12:48:02 -0600 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2014-11-19 01:52:16 -0600

Renat gravatar image

As for the first thing it's a different thread: https://ask.openstack.org/en/question...

What you wrote about misrouting RPC calls is really interesting. Let us work through it and get back to you. We've never seen anything like that even though we use Mistral with Heat and other services all the time.

In the meantime, can you please let us know which API version of Mistral are you using? v1 or v2?


edit flag offensive delete link more


Hi Renat. As soon as I observe this happening again, I will post some additional information, specifically from the rabbit queues.

dt.turner gravatar imagedt.turner ( 2014-11-20 16:57:24 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-11-13 12:20:34 -0600

Seen: 441 times

Last updated: Nov 17 '14