heat stack-list - Unexpected error occurred serving API: Timed out waiting for a reply to message ID 7daf49ca17cd4070bcfa1041890f84ae

asked 2016-06-27 01:38:38 -0500

kevin_zhu gravatar image

We came across an issue when execute 'heat stack-list' and by investigating the log of Heat api we saw the error message like below "Unexpected error occurred serving API: Timed out waiting for a reply to message ID 7daf49ca17cd4070bcfa1041890f84ae" ---->

#

2016-06-22 14:14:33.868 15417 DEBUG heat.common.serializers [req-d8ee1b0a-6568-4482-a177-e8b76356d375 - 69a1bf21012b4311b01d3f998ca95934-0c61356e-f17d-4b8d-ac33-64279d0] JSON response : {"explanation": "The server has ei ther erred or is incapable of performing the requested operation.", "code": 500, "error": {"message": "Timed out waiting for a reply to message ID 39cea53fd8de46b2a8332c32e047df59", "traceback": "Traceback (most recent c all last):\n File \"/usr/lib/python2.7/site-packages/heat/api/middleware/fault.py\", line 156, in process_request\n return req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/webob/request.p y\", line 1296, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1260, in call_application\n app_iter = application(self.environ, start_response)\n F ile \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, args, *self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n re turn self.func(req, args, *kwargs)\n File \"/usr/lib/python2.7/site-packages/heat/common/wsgi.py\", line 390, in __call__\n response = req.get_response(self.application)\n File \"/usr/lib/python2.7/site-packages/w ebob/request.py\", line 1296, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1260, in call_application\n app_iter = application(self.environ, start_ response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, args, *self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 195, in cal l_func\n return self.func(req, args, *kwargs)\n File \"/usr/lib/python2.7/site-packages/heat/common/wsgi.py\", line 390, in __call__\n response = req.get_response(self.application)\n File \"/usr/lib/python2.7/s ite-packages/webob/request.py\", line 1296, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1260, in call_application\n app_iter = application(self.e nviron, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, args, *self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", li ne 195, in call_func\n return self.func(req, args, *kwargs)\n File \"/usr/lib/python2.7/site-packages/osprofiler/web.py\", line 99, in __call__\n return request.get_response(self.application)\n File \"/usr/lib/ python2.7/site-packages/webob/request.py\", line 1296, in send\n application, catch_exc_info=False)\n File \"/usr/lib/python2.7/site-packages/webob/request.py\", line 1260, in call_application\n app_iter = applica tion(self.environ, start_response)\n File \"/usr/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, args, *self.kwargs)\n File \"/usr/lib/python2.7/site-packages/webob/d ec.py\", line 195, in ... (more)

edit retag flag offensive close merge delete

Comments

See if you have a bunch of messages sitting in the Rabbit Queues:

rabbitmqctl list_queues |grep -vi "0$"

If there are a bunch of heat queues that are sitting then verify that each heat service is running on the proper nodes.

mpetason gravatar imagempetason ( 2016-06-29 16:27:43 -0500 )edit