Alarm Trigger Delay

Alarm Trigger Delay

asked 2014-09-23 09:29:37 -0600

PatRock
40 ●8 ●11 ●16

updated 2014-10-02 08:16:49 -0600

I have a complete loadbalanced ScalingGroup, but the time it takes to create or delete a new instance takes 10 minutes.

   Policy:
     type: OS::Heat::ScalingPolicy
     properties:
       adjustment_type: change_in_capacity
       auto_scaling_group_id: {get_resource: myServer_Group}
       cooldown: 30
       scaling_adjustment: 1

   Alarm:
     type: OS::Ceilometer::Alarm
     properties:
       meter_name: cpu_util
       statistic: avg
       period: 30
       evaluation_periods: 1
      threshold: 50
    alarm_actions:
     - {get_attr: [Policy, alarm_url]}
    comparison_operator: gt

So I set the Policy Cooldown to 30seconds and the period of the alarm also to 30seconds. I keep my Instance at 90% CPU so that the alarm should trigger within the first (or second) Period. Why does it stil take 10 minutes to trigger?

I hope that someone can explain me this events and thanks to everyone who help me with my Questions.

 ceilometer alarm-history -a 5507767f-1b31-461d-a6c1-9a855ebbdbb0

 +------------------+----------------------------+--------------------------------------------+
 | Type             | Timestamp                  | Detail                                     |
 +------------------+----------------------------+--------------------------------------------+
 | creation         | 2014-10-02T12:53:02.986000 | name: Scal-cpu_alarm_HIGH-zjhgpr3ct2qy     |
 |                  |                            | type: threshold                            |
 |                  |                            | rule: cpu_util > 50.0 during 1 x 60s       |
 | state transition | 2014-10-02T13:03:35.944000 | state: alarm                               |
 | state transition | 2014-10-02T13:05:35.942000 | state: insufficient data                   |
 | state transition | 2014-10-02T13:13:36.008000 | state: alarm                               |
 | state transition | 2014-10-02T13:15:36.004000 | state: insufficient data                   |
 +------------------+----------------------------+--------------------------------------------+

answered 2014-09-23 10:13:32 -0600

larsks
5599 ●24 ●61 ●101 http://redhat.com/

updated 2014-09-23 10:14:28 -0600

I think this has to do with the default Ceilometer configuration in /etc/ceilometer/pipeline.yaml, which sets a default interval of 600 (i.e., 10 minutes) on all meters. I resolved this locally by changing the interval for the cpu_source entry:

- name: cpu_source
  interval: 60
  meters:
      - "cpu"
  sinks:
      - cpu_sink

This gets me an update once/minute. I'm not sure you want update more often than that; there may be performance consequences (or not! I don't actually know for sure).

Note that the cpu_source entry here feeds into the cpu_util metric:

- name: cpu_sink
  transformers:
      - name: "rate_of_change"
        parameters:
            target:
                name: "cpu_util"
                unit: "%"
                type: "gauge"
                scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"

edit flag offensive delete link

Comments

Thx a lot. For the performance i see in my Stack-events that the Scale_Down_Policy triggers even when the AutoScalingGroup is at the minimal size, but doesn't delete the instance. Is it possible that the alarm and policy doesn't trigger when the AutoScalingGroup is on max_size or min_size?

PatRock ( 2014-09-24 03:02:27 -0600 )edit

Which service do I need to restart to apply these changes?

PatRock ( 2014-09-24 03:32:11 -0600 )edit

I would just restart all the ceilometer services. I'm not sure which one reads this file...I would guess maybe ceilometer-collector, but I haven't tested that.

larsks ( 2014-09-24 09:40:54 -0600 )edit

This needs to be maked as answered.

asalkeld ( 2014-09-29 01:00:28 -0600 )edit

Unfortunately this doesn't seem to solve the problem :/ I changed the interval in the /etc/ceilometer/pipeline.yaml and restartet all ceilometer-services but it's still up to 10 minutes till the alarm triggers. I updated the question with the alarm-history

PatRock ( 2014-10-13 02:31:44 -0600 )edit

add a comment

1 answer

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Alarm Trigger Delay edit

1 answer

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Alarm Trigger Delay