How do I recreate every resource in an AutoScalingGroup automatically during rolling_updates?

asked 2017-04-10 17:28:56 -0500

philsttr gravatar image

My end goal is to have an OS::Heat::AutoScalingGroup with rolling updates, where the rollout process will wait for each server to complete its cloud-init before moving onto the next (to prevent downtime).

I'd also like each server to have some random data provided by a OS::Heat::RandomString.

To accomplish this, I have tried creating an OS::Heat::AutoScalingGroup with the following resources:

  • OS::Nova::Server
  • OS::Neutron::LBaaS::PoolMember (adds the server to a OS::Neutron::LBaaS::Pool)
  • OS::Heat::SoftwareConfig (used be the server, does some stuff, and signals the WaitConditionHandle)
  • OS::Heat::RandomString (used by the software config)
  • OS::Heat::WaitCondition
  • OS::Heat::WaitConditionHandle (signaled by the server after cloud-init finishes)

The AutoScalingGroup has

  rolling_updates:
    max_batch_size: 1
    min_in_service: 1

When I first create the stack, everything works great. Each server gets a new/different RandomString, the WaitConditions are waited on appropriately, etc.

However, I've noticed that when I update the stack after modifying the SoftwareConfig, only the SoftwareConfig, Server, and PoolMember resources are recreated. The RandomString, WaitCondition, and WaitConditionHandle are not recreated.

This has a couple problems:

  1. The RandomString for the new servers are the same as the old servers.
  2. The WaitCondition doesn't work. Since they aren't recreated, and were successfully triggered when they were first created, the rolling update doesn't "wait" for the condition again before moving onto the next set. This effectively renders the rolling_update useless, since it takes down all the servers before the new ones have completed initializing, which causes downtime.

Is there a way to tell the AutoScalingGroup to recreate all resources in a set when any of the resources are changed? i.e. for each "set" of those 6 resources I mentioned above, I want all 6 of those resources created, instead of only recreating 3 of them, and reusing the previous 3.

If not, is there any other way to accomplish rolling_updates in an ASG while waiting on cloud-init to finish before moving onto the next server? (to prevent any downtime)

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2017-04-17 08:56:41 -0500

zaneb gravatar image

I believe the WaitCondition problem can be resolved using the OS::Heat::UpdateWaitConditionHandle resource instead of OS::Heat::WaitConditionHandle.

For the RandomString, you probably want to change the salt each time you update.

edit flag offensive delete link more

Comments

@philsttr Did you ever find a solution for this? I'm running into the same issue.

@zaneb I tried the UpdateWaitConditionHandle, but can't figure out how to trigger the handle. The resource doesn't provide an endpoint or curl_cli attribute like the WaitConditionHandle.

DanielH gravatar imageDanielH ( 2019-10-23 17:12:28 -0500 )edit

Ah yeah, I forgot that OS::Heat::UpdateWaitConditionHandle is actually a variant on AWS::CloudFormation::WaitConditionHandle (not OS::Heat::WaitConditionHandle). So you use it using get_resource (instead of get_attr), but you may need to use an AWS::CloudFormation::WaitCondition with it.

zaneb gravatar imagezaneb ( 2019-10-24 12:59:47 -0500 )edit

Thanks for the response. I figured the get_resource part out right after I posted my last comment, but now I'm getting the "User is not authorized to perform action" error described https://ask.openstack.org/en/question/52869/heat-autoscaling-webhook-authorization-failure/ (here). Any ideas? Thx

DanielH gravatar imageDanielH ( 2019-10-25 14:25:51 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2017-04-10 17:28:56 -0500

Seen: 197 times

Last updated: Apr 17 '17