Ask Your Question
0

Alarm Trigger Delay

asked 2014-09-23 09:29:37 -0500

PatRock gravatar image

updated 2014-10-02 08:16:49 -0500

I have a complete loadbalanced ScalingGroup, but the time it takes to create or delete a new instance takes 10 minutes.

   Policy:
     type: OS::Heat::ScalingPolicy
     properties:
       adjustment_type: change_in_capacity
       auto_scaling_group_id: {get_resource: myServer_Group}
       cooldown: 30
       scaling_adjustment: 1

   Alarm:
     type: OS::Ceilometer::Alarm
     properties:
       meter_name: cpu_util
       statistic: avg
       period: 30
       evaluation_periods: 1
      threshold: 50
    alarm_actions:
     - {get_attr: [Policy, alarm_url]}
    comparison_operator: gt

So I set the Policy Cooldown to 30seconds and the period of the alarm also to 30seconds. I keep my Instance at 90% CPU so that the alarm should trigger within the first (or second) Period. Why does it stil take 10 minutes to trigger?

I hope that someone can explain me this events and thanks to everyone who help me with my Questions.

 ceilometer alarm-history -a 5507767f-1b31-461d-a6c1-9a855ebbdbb0

 +------------------+----------------------------+--------------------------------------------+
 | Type             | Timestamp                  | Detail                                     |
 +------------------+----------------------------+--------------------------------------------+
 | creation         | 2014-10-02T12:53:02.986000 | name: Scal-cpu_alarm_HIGH-zjhgpr3ct2qy     |
 |                  |                            | type: threshold                            |
 |                  |                            | rule: cpu_util > 50.0 during 1 x 60s       |
 | state transition | 2014-10-02T13:03:35.944000 | state: alarm                               |
 | state transition | 2014-10-02T13:05:35.942000 | state: insufficient data                   |
 | state transition | 2014-10-02T13:13:36.008000 | state: alarm                               |
 | state transition | 2014-10-02T13:15:36.004000 | state: insufficient data                   |
 +------------------+----------------------------+--------------------------------------------+
edit retag flag offensive close merge delete
add a comment

1 answer

Sort by ยป oldest newest most voted
2

answered 2014-09-23 10:13:32 -0500

larsks gravatar image

updated 2014-09-23 10:14:28 -0500

I think this has to do with the default Ceilometer configuration in /etc/ceilometer/pipeline.yaml, which sets a default interval of 600 (i.e., 10 minutes) on all meters. I resolved this locally by changing the interval for the cpu_source entry:

- name: cpu_source
  interval: 60
  meters:
      - "cpu"
  sinks:
      - cpu_sink

This gets me an update once/minute. I'm not sure you want update more often than that; there may be performance consequences (or not! I don't actually know for sure).

Note that the cpu_source entry here feeds into the cpu_util metric:

- name: cpu_sink
  transformers:
      - name: "rate_of_change"
        parameters:
            target:
                name: "cpu_util"
                unit: "%"
                type: "gauge"
                scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
edit flag offensive delete link more

Comments

Thx a lot. For the performance i see in my Stack-events that the Scale_Down_Policy triggers even when the AutoScalingGroup is at the minimal size, but doesn't delete the instance. Is it possible that the alarm and policy doesn't trigger when the AutoScalingGroup is on max_size or min_size?

PatRock gravatar imagePatRock ( 2014-09-24 03:02:27 -0500 )edit

Which service do I need to restart to apply these changes?

PatRock gravatar imagePatRock ( 2014-09-24 03:32:11 -0500 )edit

I would just restart all the ceilometer services. I'm not sure which one reads this file...I would guess maybe ceilometer-collector, but I haven't tested that.

larsks gravatar imagelarsks ( 2014-09-24 09:40:54 -0500 )edit

This needs to be maked as answered.

asalkeld gravatar imageasalkeld ( 2014-09-29 01:00:28 -0500 )edit

Unfortunately this doesn't seem to solve the problem :/ I changed the interval in the /etc/ceilometer/pipeline.yaml and restartet all ceilometer-services but it's still up to 10 minutes till the alarm triggers. I updated the question with the alarm-history

PatRock gravatar imagePatRock ( 2014-10-13 02:31:44 -0500 )edit
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2014-09-23 09:29:37 -0500

Seen: 229 times

Last updated: Oct 02 '14

Related questions

heat template create multiple instances

Floating Loadbalancing IP

heat variable substitution

heat check if resource exists before creating it

Template Parameter label doesn't apply

Heat autoscallin issue

Is it possible to launch instance based on parameter passed to the heat template

Error : Unable to find port with name or id

Timeout on Heat while creating a Sahara cluster

Setting a default locale