Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Autoscaling -Exception during message handling

Hi, I'm using HOT autoscaling with ceilometer alarms. Scale down and scale up policies were successfully triggered, but when my scaling group meet my group minimum size I'm getting the attached error. Ceilometer works fine and Rabbitmq configuration is the same as Heat.

/var/log/heat/heat-engine.log:

2014-07-17 13:46:16.062 8623 ERROR heat.openstack.common.rpc.amqp [req-87797472-dbca-4298-ab42-fd960c9a8cc5 None] Exception during mesage handling
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp Traceback (most recent call last):
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/openstack/common/rpc/mqp.py", line 462, in _process_data
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     **args)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/openstack/common/rpc/ispatcher.py", line 172, in dispatch
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", lne 63, in wrapped
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     return func(self, ctx, *args, **kwargs)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", lne 890, in resource_signal
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     stack_context = self.load_user_creds(s.user_creds_id)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", lne 308, in load_user_creds
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     hkc.KeystoneClient(stored_context)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystonecient.py", line 529, in __new__
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     return KeystoneClientV3(context)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystonecient.py", line 81, in __init__
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     self._client = self._v3_client_init()
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystonecient.py", line 166, in _v3_client_init
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     client.authenticate()
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", ine 318, in inner
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     return func(*args, **kwargs)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.y", line 392, in authenticate
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     resp = self.get_raw_token_from_identity_service(**kwargs)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/client.p", line 169, in get_raw_token_from_identity_service
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     return a.get_auth_ref(self.session)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identiy/v3.py", line 110, in get_auth_ref
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     authenticated=False)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py" line 310, in post
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     return self.request(url, 'POST', **kwargs)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", ine 318, in inner
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     return func(*args, **kwargs)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py" line 251, in request
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp     raise exceptions.from_response(resp, method, url)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp Unauthorized: The request you have made requires authentication. (HTP 401)
2014-07-17 13:46:16.062 8623 TRACE heat.openstack.common.rpc.amqp
2014-07-17 13:46:16.065 8623 ERROR heat.openstack.common.rpc.common [req-87797472-dbca-4298-ab42-fd960c9a8cc5 None] Returning exceptin The request you have made requires authentication. (HTTP 401) to caller
2014-07-17 13:46:16.065 8623 ERROR heat.openstack.common.rpc.common [req-87797472-dbca-4298-ab42-fd960c9a8cc5 None] ['Traceback (mostrecent call last):\n', '  File "/usr/lib/python2.7/dist-packages/heat/openstack/common/rpc/amqp.py", line 462, in _process_data\n    *args)\n', '  File "/usr/lib/python2.7/dist-packages/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = geattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 63, in wrapped\n   return func(self, ctx, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 890, in resourcesignal\n    stack_context = self.load_user_creds(s.user_creds_id)\n', '  File "/usr/lib/python2.7/dist-packages/heat/engine/service.p", line 308, in load_user_creds\n    hkc.KeystoneClient(stored_context)\n', '  File "/usr/lib/python2.7/dist-packages/heat/common/hea_keystoneclient.py", line 529, in __new__\n    return KeystoneClientV3(context)\n', '  File "/usr/lib/python2.7/dist-packages/heat/comon/heat_keystoneclient.py", line 81, in __init__\n    self._client = self._v3_client_init()\n', '  File "/usr/lib/python2.7/dist-pacages/heat/common/heat_keystoneclient.py", line 166, in _v3_client_init\n    client.authenticate()\n', '  File "/usr/lib/python2.7/dis-packages/keystoneclient/utils.py", line 318, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-package/keystoneclient/httpclient.py", line 392, in authenticate\n    resp = self.get_raw_token_from_identity_service(**kwargs)\n', '  File /usr/lib/python2.7/dist-packages/keystoneclient/v3/client.py", line 169, in get_raw_token_from_identity_service\n    return a.get_aut_ref(self.session)\n', '  File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3.py", line 110, in get_auth_ref\n    uthenticated=False)\n', '  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 310, in post\n    return self.requst(url, \'POST\', **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner\n    return fuc(*args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 251, in request\n    raise exceptios.from_response(resp, method, url)\n', 'Unauthorized: The request you have made requires authentication. (HTTP 401)\n']

/var/log/ceilometer/ceilometer-alarm-evaluator.log:

2014-07-17 13:38:01.393 12031 INFO ceilometer.alarm.evaluator [-] alarm eba4985c-2da1-4d0e-98b5-d45982071a9d transitioning to alarm because Transition to alarm due to 1 samples outside threshold, most recent: 99.0081967213
2014-07-17 13:38:01.520 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 23.902322 sec
2014-07-17 13:38:01.897 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:39:23.256 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by     21.734303 sec
2014-07-17 13:39:23.279 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:40:04.318 12031 INFO ceilometer.alarm.evaluator [-] alarm c0e15308-81d2-4d63-b149-42a6f231113b transitioning to ok because Transition to ok due to 1 samples inside threshold, most recent: 18.0853191543
2014-07-17 13:40:45.331 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.073905 sec
2014-07-17 13:40:45.350 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:42:07.783 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.450842 sec
2014-07-17 13:42:07.800 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:43:30.437 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by  22.651982 sec
2014-07-17 13:43:30.801 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:44:53.253 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.815025 sec
2014-07-17 13:44:53.275 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:46:15.504 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.250391 sec
2014-07-17 13:46:15.524 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:47:39.023 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 23.517472 sec
2014-07-17 13:47:39.045 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:49:01.084 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.059401 sec
2014-07-17 13:49:01.430 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:50:23.691 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.60513 sec
2014-07-17 13:50:23.709 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms
2014-07-17 13:51:45.885 12031 WARNING ceilometer.openstack.common.loopingcall [-] task run outlasted interval by 22.19314 sec
2014-07-17 13:51:45.905 12031 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 2 alarms