Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Mistral getting sent requests meant for heat endpoint

Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from github based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:

http://paste.openstack.org/show/132900/

I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:

[keystone_authtoken]

auth_host = 10.241.70.70

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx

auth_uri= http://10.241.70.70:5000/v2.0

Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on.

Mistral getting sent requests meant for heat endpoint

Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from github based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:

http://paste.openstack.org/show/132900/

I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:

[keystone_authtoken]

auth_host = 10.241.70.70

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx

auth_uri= http://10.241.70.70:5000/v2.0

Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on.

Update: OK.. so things just got a bit weirder. I deleted the mistral service and endpoint from keystone and bounced the keystone service. I left mistral server running within it's VM and can still see that heat requests are getting routed to it via the log errors.

Mistral getting sent requests meant for heat endpoint

Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from github based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:

http://paste.openstack.org/show/132900/

I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:

[keystone_authtoken]

auth_host = 10.241.70.70

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx

auth_uri= http://10.241.70.70:5000/v2.0

Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on.

Update: OK.. so things just got a bit weirder. I deleted the mistral service and endpoint from keystone and bounced the keystone service. I left mistral server running within it's VM and can still see that heat requests are getting routed to it via the log errors.

Update: Added the mistral keystone service and endpoint back in order to see if the problem somehow had to do with the original service / endpoint entries, but still seeing the same behavior. I reported earlier that I'm able to perform mistral list actions, however that is only from the mistral node (running in an instance with associated floating ip). If I try mistral list actions from another host, I get: ('Connection aborted.', error(111, 'Connection refused'))

I verified that the VM has the mistral api service port 8989 open via: nc [mistral_node_ip] 8989; echo $?]

If I run heat api calls, every other call gets punted to the mistral service node instead of the heat node:

First attempt OK:

heat stack-list +--------------------------------------+---------------------+-----------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+---------------------+-----------------+----------------------+ | 2dcbb008-406e-4553-9a31-daa218d08a51 | test-stack1 | CREATE_FAILED | 2014-11-12T22:07:36Z | | 90c0caae-aef7-4191-b6d5-1abc69c93e15 | test-stack2 | CREATE_COMPLETE | 2014-11-13T01:19:18Z | +--------------------------------------+---------------------+-----------------+----------------------+

Second attempt, NOT OK: heat stack-list ERROR: Remote error: NoSuchMethod Endpoint does not support RPC method list_stacks

After this second attempt, you see in mistral logs (on a completely different endpoint ip and port!):

2014-11-13 15:41:59.874 3965 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Endpoint does not support RPC method list_stacks 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 184, in _dispatch 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher raise NoSuchMethod(method) 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher NoSuchMethod: Endpoint does not support RPC method list_stacks 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher 2014-11-13 15:41:59.876 3965 ERROR oslo.messaging._drivers.common [-] Returning exception Endpoint does not support RPC method list_stacks to caller 2014-11-13 15:41:59.876 3965 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 184, in _dispatch\n raise NoSuchMethod(method)\n', 'NoSuchMethod: Endpoint does not support RPC method list_stacks\n']

Mistral getting sent requests meant for heat API calls routed to wrong service endpoint

Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from github based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:

http://paste.openstack.org/show/132900/

I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:

[keystone_authtoken]

auth_host = 10.241.70.70

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx

auth_uri= http://10.241.70.70:5000/v2.0

Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on.

Update: OK.. so things just got a bit weirder. I deleted the mistral service and endpoint from keystone and bounced the keystone service. I left mistral server running within it's VM and can still see that heat requests are getting routed to it via the log errors.

Update: Added the mistral keystone service and endpoint back in order to see if the problem somehow had to do with the original service / endpoint entries, but still seeing the same behavior. I reported earlier that I'm able to perform mistral list actions, however that is only from the mistral node (running in an instance with associated floating ip). If I try mistral list actions from another host, I get: ('Connection aborted.', error(111, 'Connection refused'))

I verified that the VM has the mistral api service port 8989 open via: nc [mistral_node_ip] 8989; echo $?]

If I run heat api calls, every other call gets punted to the mistral service node instead of the heat node:

First attempt OK:

heat stack-list +--------------------------------------+---------------------+-----------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+---------------------+-----------------+----------------------+ | 2dcbb008-406e-4553-9a31-daa218d08a51 | test-stack1 | CREATE_FAILED | 2014-11-12T22:07:36Z | | 90c0caae-aef7-4191-b6d5-1abc69c93e15 | test-stack2 | CREATE_COMPLETE | 2014-11-13T01:19:18Z | +--------------------------------------+---------------------+-----------------+----------------------+

Second attempt, NOT OK: heat stack-list ERROR: Remote error: NoSuchMethod Endpoint does not support RPC method list_stacks

After this second attempt, you see in mistral logs (on a completely different endpoint ip and port!):

2014-11-13 15:41:59.874 3965 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Endpoint does not support RPC method list_stacks 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 184, in _dispatch 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher raise NoSuchMethod(method) 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher NoSuchMethod: Endpoint does not support RPC method list_stacks 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher 2014-11-13 15:41:59.876 3965 ERROR oslo.messaging._drivers.common [-] Returning exception Endpoint does not support RPC method list_stacks to caller 2014-11-13 15:41:59.876 3965 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 184, in _dispatch\n raise NoSuchMethod(method)\n', 'NoSuchMethod: Endpoint does not support RPC method list_stacks\n']

API calls routed to wrong service endpoint

Hello all. I've got myself a strange one here....

My team owns an Icehouse 2014.1.2 based cloud. We have 1 dedicated control node, a network node, and a DB node. 6 compute nodes are currently active.

I've spun up a VM within the cloud and installed mistral from github based on the setup steps outlined there. I installed the latest version as of Monday, November 10 2014.

After completing the installation steps, I've verified that the mistral DB and tables have been created. The mistral services successfully checkin with the rabbitmq server running on our control node. I can perform list actions, ie: workbook-list, action-list, etc.

What I cannot do is create any workbooks. Now, this may be a separate issue from the question this thread aims to ask, but I'll mention it in case they are related. Upon trying to add the example workbook for vm creation from mistral-extra-0.1.1/examples/v1/create_vm, I get the following error in mistral logs:

http://paste.openstack.org/show/132900/

I've sourced my environment and can successfully run nova list and create actions against my Icehouse endpoint from the Mistral virtualenv where Mistral services are running. My mistral.conf has the following defined:

[keystone_authtoken]

auth_host = 10.241.70.70

auth_port = 35357

auth_protocol = http

admin_tenant_name = services

admin_user = mistral

admin_password = xxxxxxxx

auth_uri= http://10.241.70.70:5000/v2.0

Now here's the really "interesting" part. While watching the mistral logs for clues, I noticed noSuchMethod TRACE messages showing up. Upon closer examination, I saw that the requests were for heat stack related methods. At the same time a team member pinged me to tell me that their heat requests were timing out about every other attempt. When they saw time outs, I saw the request errors in the mistral logs.

My first thought what that the endpoints were misconfigured, but they are not. The heat api and engine services are running on the control node, while the mistral services are running in a VM with a completely different IP and port. The keystone service endpoints reflect this: http://paste.openstack.org/show/132568/

I've bounced heat and keystone services but this had no effect. When I stop mistral services, all heat api calls get properly routed to the correct endpoint.

I'm currently at a loss as to what's going on.on. Any general tips for troubleshooting incorrectly routed API calls outside of verifying the keystone service endpoint configuration?

Update: OK.. so things just got a bit weirder. I deleted the mistral service and endpoint from keystone and bounced the keystone service. I left mistral server running within it's VM and can still see that heat requests are getting routed to it via the log errors.

Update: Added the mistral keystone service and endpoint back in order to see if the problem somehow had to do with the original service / endpoint entries, but still seeing the same behavior. I reported earlier that I'm able to perform mistral list actions, however that is only from the mistral node (running in an instance with associated floating ip). If I try mistral list actions from another host, I get: ('Connection aborted.', error(111, 'Connection refused'))

I verified that the VM has the mistral api service port 8989 open via: nc [mistral_node_ip] 8989; echo $?]

If I run heat api calls, every other call gets punted to the mistral service node instead of the heat node:

First attempt OK:

heat stack-list +--------------------------------------+---------------------+-----------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+---------------------+-----------------+----------------------+ | 2dcbb008-406e-4553-9a31-daa218d08a51 | test-stack1 | CREATE_FAILED | 2014-11-12T22:07:36Z | | 90c0caae-aef7-4191-b6d5-1abc69c93e15 | test-stack2 | CREATE_COMPLETE | 2014-11-13T01:19:18Z | +--------------------------------------+---------------------+-----------------+----------------------+

Second attempt, NOT OK: heat stack-list ERROR: Remote error: NoSuchMethod Endpoint does not support RPC method list_stacks

After this second attempt, you see in mistral logs (on a completely different endpoint ip and port!):

2014-11-13 15:41:59.874 3965 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Endpoint does not support RPC method list_stacks 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 184, in _dispatch 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher raise NoSuchMethod(method) 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher NoSuchMethod: Endpoint does not support RPC method list_stacks 2014-11-13 15:41:59.874 3965 TRACE oslo.messaging.rpc.dispatcher 2014-11-13 15:41:59.876 3965 ERROR oslo.messaging._drivers.common [-] Returning exception Endpoint does not support RPC method list_stacks to caller 2014-11-13 15:41:59.876 3965 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/root/mistral/mistral-0.1.1/.tox/venv/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 184, in _dispatch\n raise NoSuchMethod(method)\n', 'NoSuchMethod: Endpoint does not support RPC method list_stacks\n']