Ask Your Question
3

Heat Orchestration: Scale-down a specific instance

asked 2015-01-16 10:53:05 -0500

bigno gravatar image

Hi,

this is my OpenStack configuration (Nova Legacy/nova-network):

  • 1 Controller Node (with Identity, Image Service, Compute, Dashboard, Orchestration and Telemetry)
  • 1 Compute Node (with Compute and Telemetry)

I'm trying to create a auto-scaling behavior for a Web service. For the realization of auto-scaling I made use of the templates offered by the component Heat for orchestration of OpenStack.

In particular, I used the following template:

heat_template_version: 2013-05-23

description: >
  15 Gennaio 2015
  First HOT (Heat Orchestration Template)

parameters:
  key_name:
    type: string
    description: Name of an existing key pair to use for the instances
    default: admin-key-2015-2
    constraints:
      - custom_constraint: nova.keypair
        description: Must name a public key (pair) known to Nova
  flavor:
    type: string
    description: Flavor for the instances to be created
    default: custom.Flavor2 # 4 vCPUs - 2 GB RAM - 10 GB Disk
    constraints:
      - custom_constraint: nova.flavor
        description: Must be a flavor known to Nova
  image:
    type: string
    description: Name or ID of the image to use for the instances.
    default: UbuntuImage-SdEWebService #Ubuntu-14.04-CloudImage
    constraints:
      - custom_constraint: glance.image
        description: Must identify an image known to Glance
  netID:
    type: string
    description: The network for the VM
    default: 820a1c5b-a8d2-48a4-a755-45eade9a3d5e

resources:
  asg:
    type: OS::Heat::AutoScalingGroup
    properties:
      resource:
        type: OS::Nova::Server
        properties:
          key_name: { get_param: key_name }
          image: { get_param: image }
          flavor: { get_param: flavor }
          networks: [{network: {get_param: netID} }]
          metadata: {"metering.stack": {get_param: "OS::stack_id"}}
      min_size: 1
      desired_capacity: 1
      max_size: 3

  scale_up_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: asg}
      cooldown: 60
      scaling_adjustment: 1

  scale_down_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: asg}
      cooldown: 60
      scaling_adjustment: '-1'

  cpu_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average CPU > 50% for 1 minute
      meter_name: cpu_util
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 50
      alarm_actions:
        - {get_attr: [scale_up_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: gt

  cpu_alarm_low:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-down if the average CPU < 15% for 10 minutes
      meter_name: cpu_util
      statistic: avg
      period: 600
      evaluation_periods: 1
      threshold: 15
      alarm_actions:
        - {get_attr: [scale_down_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: lt

outputs:
  scale_up_url:
    description: >
      This URL is the webhook to scale up the group.  You can invoke
      the scale-up operation by doing an HTTP POST to this URL; no
      body nor extra headers are needed.
    value: {get_attr: [scale_up_policy, alarm_url]}
  scale_dn_url:
    description: >
      This URL is the webhook to scale down the group.  You can invoke
      the scale-down operation by doing an HTTP POST to this URL; no
      body nor extra headers are needed.
    value: {get_attr: [scale_down_policy, alarm_url]}
  ceilometer_query:
    value:
      str_replace:
        template: >
          ceilometer statistics -m cpu_util
          -q metadata.user_metadata.stack=stackval -p 600 -a avg
        params:
          stackval: { get_param: "OS::stack_id" }
    description: >
      This is a Ceilometer query for statistics on the cpu_util meter
      Samples about OS::Nova::Server instances in this stack.  The -q
      parameter selects Samples according to the subject's metadata.
      When a VM's metadata includes an item of the form metering.X=Y,
      the corresponding Ceilometer resource has a metadata item of the
      form user_metadata.X=Y and samples about resources so tagged can
      be queried with a ...
(more)
edit retag flag offensive close merge delete

5 answers

Sort by ยป oldest newest most voted
1

answered 2015-02-01 04:06:32 -0500

Qiming gravatar image

Thanks for bringing up this again. Scaling down policy has long time been a problem to solve. Team is working on solving this.

edit flag offensive delete link more
1

answered 2015-05-06 11:34:51 -0500

cduch gravatar image

If you want to use autoscaling, you have to free your mind from thinking of specific VMs( e.g. VM-1 or VM-2). In an ideal world, all members in the lb-pool should be stateless and able to answer the requests. So it really doesn't matter which member is removed. The load will balance to the rest of them and create new members/instances if necessary.

When looking at your example, you want heat to detect which instance has "more" load and remove the other one. In real life, you want to create a rule like "remove an instance if it's idling for more then n minutes". In this case, the state "idling" has to defined by yourself. Sometimes you can't use the CPU itself. e.g detect if there is a valid session which is not timed out.

In my opinion, a really intelligent mechanism for autoscaling isn't possible with heat only. So if you want a really intelligent mechanism for autoscaling, you have to build it yourself depending on your application and using the OpenStack API for creation/removal of instances.

edit flag offensive delete link more

Comments

Well, we now have the senlin service released with Mitaka. It is a clustering service (http://wiki.openstack.org/wiki/senlin). Requirements like this one can be easily met using senlin now.

Qiming gravatar imageQiming ( 2016-04-08 00:05:56 -0500 )edit
0

answered 2016-04-07 16:26:34 -0500

I have the same question. Even though the alarm state changes to OK, scaledown policy never delete a VM. Am I missing something here? Can someone help?

edit flag offensive delete link more

Comments

Never mind, I have made it working by creating another alarm. Earlier I was using "ok_actions" and scaledown_policy

kranthi.guttikonda9@gmail.com gravatar imagekranthi.guttikonda9@gmail.com ( 2016-04-07 16:34:37 -0500 )edit

Well, we now have the senlin service released with Mitaka. It is a clustering service (http://wiki.openstack.org/wiki/senlin). Requirements like this one can be easily met using senlin now.

Qiming gravatar imageQiming ( 2016-04-08 00:07:02 -0500 )edit
0

answered 2015-05-06 06:53:40 -0500

swati-shukla1 gravatar image

What is the status of scale down policy currently?

edit flag offensive delete link more

Comments

Well, we now have the senlin service released with Mitaka. It is a clustering service (http://wiki.openstack.org/wiki/senlin). Requirements like this one can be easily met using senlin now.

Qiming gravatar imageQiming ( 2016-04-08 00:06:09 -0500 )edit
0

answered 2016-05-19 16:11:13 -0500

zaneb gravatar image

From Mitaka onward, you can use the mark-unhealthy command to mark a particular resource as unhealthy. This will put it first in line to get removed if there is a subsequent scale-down. Even if there isn't, it will get replaced the next time the scaling group is updated - so on the next scale up, or even just doing a stack update without modifying anything.

edit flag offensive delete link more

Comments

Hi, resource-mark-unhealthy option available to only resources listed in heat resource-list <stack_id>. But I wanted to mark specific VM as unhealthy, so that I can remove specific VM during scaledown operation. Thanks in advance.

Reddeuah Raju K gravatar imageReddeuah Raju K ( 2016-09-23 05:04:29 -0500 )edit

You just need to know the resource name that corresponds to the VM. I'm hoping to improve that in Ocata though: https://bugs.launchpad.net/heat/+bug/...

zaneb gravatar imagezaneb ( 2016-12-02 08:04:51 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2015-01-16 10:53:05 -0500

Seen: 3,168 times

Last updated: May 19 '16