Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from https://github.com/stackforge/mistral (github) based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:

http://paste.openstack.org/show/132900/

I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:

[keystone_authtoken]

auth_host = 10.241.70.70

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx

auth_uri= http://10.241.70.70:5000/v2.0

Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on. Any general tips for troubleshooting incorrectly routed API calls outside of verifying the keystone service endpoint configuration?

Update: OK.. so things just got a bit weirder. I deleted the mistral service and endpoint from keystone and bounced the keystone service. I left mistral server running within it's VM and can still see that heat requests are getting routed to it via the log errors.

Update: Added the mistral keystone service and endpoint back in order to see if the problem somehow had to do with the original service / endpoint entries, but still seeing the same ... (more)