Ask Your Question
3

Create a highly available Auto Scaling Group

asked 2014-09-17 04:41:45 -0500

erivni gravatar image

I trying to deploy a HA scaling group using AutoscalingGroup and HARestarter resources. I combined these 2 templates:

Here are my heat templates for a single HA Scaling group.

I'm using Redhat Icehouse Openststack (heat 0.2.9) with a Centos65 image.

While deploying this stack I'm getting this error regarding autoscaling ceilometer alarm :

/var/log/heat/engine.log:

2014-09-17 12:21:38.952 11734 INFO heat.engine.watchrule [req-868d212b-3e8b-4a30-98ba-39f6c5b65bf0 None] WATCH: stack:a0f6d7ca-b617-42b3-b545-826944f70021, watch_name:haautoscaling-web_server_group-avw5lve4v6pf-ew4m6jnagz7x-2vs7tvmcaomn-heartbeat_failure_alarm, new_state:NORMAL
2014-09-17 12:21:38.953 11734 INFO heat.engine.watchrule [req-868d212b-3e8b-4a30-98ba-39f6c5b65bf0 None] no action for new state NORMAL
2014-09-17 12:21:38.967 11734 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.56.165.25
2014-09-17 12:21:39.133 11734 INFO heat.engine.watchrule [req-0cc46c90-c7c1-4e54-b6ba-db05307400ad None] WATCH: stack:e17b2e84-ae5a-4dd1-9bda-fbb3c8906cf3, watch_name:haautoscaling-web_server_group-avw5lve4v6pf-4sy5bnthbvos-oipuypiu3kye-heartbeat_failure_alarm, new_state:NORMAL
 2014-09-17 12:21:39.133 11734 INFO heat.engine.watchrule [req-0cc46c90-c7c1-4e54-b6ba-db05307400ad None] no action for new state NORMAL
 2014-09-17 12:21:39.146 11734 ERROR heat.openstack.common.loopingcall [-] in fixed duration looping call
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall Traceback (most recent call last):
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/heat/openstack/common/loopingcall.py", line 76, in _inner
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall     self.f(*self.args, **self.kw)
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 240, in periodic_watcher_task
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall     self.check_stack_watches(sid)
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 229, in check_stack_watches
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall     actions = rule.evaluate()
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/heat/engine/watchrule.py", line 234, in evaluate
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall     return self.run_rule()
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/heat/engine/watchrule.py", line 241, in run_rule
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall     new_state = self.get_alarm_state()
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/heat/engine/watchrule.py", line 224, in get_alarm_state
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall     fn = getattr(self, 'do_%s' % self.rule['Statistic'])
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall KeyError: 'Statistic'
 2014-09-17 12:21:39.146 11734 TRACE heat.openstack.common.loopingcall

It seems like heat-engine trying to parse ceilometer cpu_alarm_high/low as a cloudwatch alarm. Using each feature alone - autoscaling with ceilometer and restart with cloudwatch, worked fine and never gave this error.

Is it possible to combine between those resources? Is Ceilometer going to ... (more)

edit retag flag offensive close merge delete

2 answers

Sort by » oldest newest most voted
1

answered 2014-09-28 23:19:28 -0500

asalkeld gravatar image

This looks like a bug :-(

I'll make a bug for you on launchpad.

edit flag offensive delete link more

Comments

https://bugs.launchpad.net/heat/+bug/...

Note: you can use cfn-push-stats together with Ceilometer alarms. See: https://github.com/asalkeld/autoscali...

asalkeld gravatar imageasalkeld ( 2014-09-28 23:23:43 -0500 )edit
asalkeld gravatar imageasalkeld ( 2014-09-28 23:39:18 -0500 )edit
0

answered 2014-10-02 01:32:47 -0500

erivni gravatar image

updated 2014-10-02 01:35:47 -0500

Thanks a lot! it solved the bug..

But, there's still a problem with my solution. My Ceilometer alarms are always on ‘insufficient data’ state. According to your wordpress post I guess that my problem is with ‘matching_metadata’ property.

I using the following metadata:

instance metadata:{groupname:‘myGroup'’}

alarm matching_metadata: {metadata.metering.groupname: ‘myGroup'}

but it's not working..

Also, I would like to ask if there's another way to implement a HA scaling group? Ceilometer does not support heatbeat meter so when I’m trying to use Ceilometer with cfn-push-stat it fails with KeyError: 'counter_name'. Also, I couldn’t use LaunchConfiguration resource because that way I won’t be able to restart a failed server since there is an instance_id property inside HARestarter which I can’t get from an AWS ScalingGroup resource..

Thanks again.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-09-17 04:41:45 -0500

Seen: 709 times

Last updated: Oct 02 '14