Alarm Trigger Delay

asked 2014-09-23 09:29:37 -0500

PatRock gravatar image

updated 2014-10-02 08:16:49 -0500

I have a complete loadbalanced ScalingGroup, but the time it takes to create or delete a new instance takes 10 minutes.

     type: OS::Heat::ScalingPolicy
       adjustment_type: change_in_capacity
       auto_scaling_group_id: {get_resource: myServer_Group}
       cooldown: 30
       scaling_adjustment: 1

     type: OS::Ceilometer::Alarm
       meter_name: cpu_util
       statistic: avg
       period: 30
       evaluation_periods: 1
      threshold: 50
     - {get_attr: [Policy, alarm_url]}
    comparison_operator: gt

So I set the Policy Cooldown to 30seconds and the period of the alarm also to 30seconds. I keep my Instance at 90% CPU so that the alarm should trigger within the first (or second) Period. Why does it stil take 10 minutes to trigger?

I hope that someone can explain me this events and thanks to everyone who help me with my Questions.

 ceilometer alarm-history -a 5507767f-1b31-461d-a6c1-9a855ebbdbb0

 | Type             | Timestamp                  | Detail                                     |
 | creation         | 2014-10-02T12:53:02.986000 | name: Scal-cpu_alarm_HIGH-zjhgpr3ct2qy     |
 |                  |                            | type: threshold                            |
 |                  |                            | rule: cpu_util > 50.0 during 1 x 60s       |
 | state transition | 2014-10-02T13:03:35.944000 | state: alarm                               |
 | state transition | 2014-10-02T13:05:35.942000 | state: insufficient data                   |
 | state transition | 2014-10-02T13:13:36.008000 | state: alarm                               |
 | state transition | 2014-10-02T13:15:36.004000 | state: insufficient data                   |
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2014-09-23 10:13:32 -0500

larsks gravatar image

updated 2014-09-23 10:14:28 -0500

I think this has to do with the default Ceilometer configuration in /etc/ceilometer/pipeline.yaml, which sets a default interval of 600 (i.e., 10 minutes) on all meters. I resolved this locally by changing the interval for the cpu_source entry:

- name: cpu_source
  interval: 60
      - "cpu"
      - cpu_sink

This gets me an update once/minute. I'm not sure you want update more often than that; there may be performance consequences (or not! I don't actually know for sure).

Note that the cpu_source entry here feeds into the cpu_util metric:

- name: cpu_sink
      - name: "rate_of_change"
                name: "cpu_util"
                unit: "%"
                type: "gauge"
                scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
edit flag offensive delete link more


Thx a lot. For the performance i see in my Stack-events that the Scale_Down_Policy triggers even when the AutoScalingGroup is at the minimal size, but doesn't delete the instance. Is it possible that the alarm and policy doesn't trigger when the AutoScalingGroup is on max_size or min_size?

PatRock gravatar imagePatRock ( 2014-09-24 03:02:27 -0500 )edit

Which service do I need to restart to apply these changes?

PatRock gravatar imagePatRock ( 2014-09-24 03:32:11 -0500 )edit

I would just restart all the ceilometer services. I'm not sure which one reads this file...I would guess maybe ceilometer-collector, but I haven't tested that.

larsks gravatar imagelarsks ( 2014-09-24 09:40:54 -0500 )edit

This needs to be maked as answered.

asalkeld gravatar imageasalkeld ( 2014-09-29 01:00:28 -0500 )edit

Unfortunately this doesn't seem to solve the problem :/ I changed the interval in the /etc/ceilometer/pipeline.yaml and restartet all ceilometer-services but it's still up to 10 minutes till the alarm triggers. I updated the question with the alarm-history

PatRock gravatar imagePatRock ( 2014-10-13 02:31:44 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-09-23 09:29:37 -0500

Seen: 275 times

Last updated: Oct 02 '14