Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

testing heat autoscaling fails

I installed a devstack (heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }

testing heat autoscaling fails

I installed a devstack (heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. Any advice is most appreciated.

This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }

testing heat autoscaling fails

I installed a devstack (heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. Any advice is most appreciated.

This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm: alarm, but that there is an authentication failure that might be preventing the extra instance to be launched:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }

testing heat autoscaling failsaws authentication failure

I installed a devstack (heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. Any advice is most appreciated.

This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm, but that there is an authentication failure that might be preventing the extra instance to be launched:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }

heat autoscaling aws authentication failure

I installed a devstack devstack, stable/havana (heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. Any advice is most appreciated.

This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm, but that there is an authentication failure that might be preventing the extra instance to be launched:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }

heat autoscaling aws authentication failure

I installed devstack, stable/havana (heat (single node, stable/havana, heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. Any advice is most appreciated.

This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm, but that there is an authentication failure that might be preventing the extra instance to be launched:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }

heat autoscaling aws authentication failure

I installed devstack, (single node, stable/havana, heat and ceilometer included) and was hoping to test the autoscaling feature. I am using a custom meter ("testcounter") in ceilometer. The meter seems to be working fine, I post samples to trigger the alarm using the REST API.

When I launch the stack, one instance is booted. But when the alarm is triggered, no additional instance is booting.. Any advice is most appreciated.

This is the h-api-cfn output, which seems to indicate that heat receives the ceilometer alarm, but that there is an authentication failure that might be preventing the extra instance to be launched:

2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::574192af441e4e0487b0ad60a9fda946:stacks/test/f04974cc-3d64-4fc1-ab0c-04518a595477/resources/WebServerScaleDownPolicy Accept: */* from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:53
2014-02-26 13:41:47.776 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=21822) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:68
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-02-26 13:41:47.777 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-02-26 13:41:47.778 INFO heat.api.aws.ec2token [-] Authenticating with http://<ip_controller>:5000/v2.0/ec2tokens
2014-02-26 13:41:47.791 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): proxy.atlantis.ugent.be
<proxy>
2014-02-26 13:41:47.803 DEBUG requests.packages.urllib3.connectionpool [-] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.1" 500 183 from (pid=21822) _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:344
2014-02-26 13:41:47.804 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-02-26 13:41:47.805 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> from (pid=21822) to_xml /opt/stack/heat/heat/common/wsgi.py:604

This is the keystone log:

2014-02-26 13:41:47.609 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/tokens HTTP/1
.0" 200 9697
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Matched POST /ec2tokens from (pid=11474) __call__ /usr/lib/python2.7/dist-packages
/routes/middleware.py:100
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Route path: '/ec2tokens', defaults: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-02-26 13:41:47.797 DEBUG routes.middleware [-] Match dict: {'action': u'authenticate', 'controller': <keystone.contrib.ec2.controllers.Ec2Controller object at 0x4518f50>} from (pid=11474) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-02-26 13:41:47.801 ERROR keystone.common.wsgi [-] 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 238, in __call__
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     result = method(context, **params)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 96, in authenticate
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     creds_ref = self._get_credentials(credentials['access'])
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 237, in _get_credentials
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     return self._convert_v3_to_ec2_credential(creds)
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/contrib/ec2/controllers.py", line 222, in _convert_v3_to_ec2_credential
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi     'access': blob.get('access'),
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi AttributeError: 'unicode' object has no attribute 'get'
2014-02-26 13:41:47.801 TRACE keystone.common.wsgi 
2014-02-26 13:41:47.802 INFO access [-] 157.193.215.17 - - [26/Feb/2014:12:41:47 +0000] "POST http://<ip_controller>:5000/v2.0/ec2tokens HTTP/1.0" 500 183

This is my heat template, notice I don't use a loadbalancer, as I assume this isn't required (?):

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample Template",

  "Parameters" : {
"KeyName" : {
  "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
  "Type" : "String"
},

"InstanceType" : {
  "Description" : "WebServer EC2 instance type",
  "Type" : "String",
  "Default" : "m1.small",
  "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
  "ConstraintDescription" : "must be a valid EC2 instance type."
},

"DBName": {
  "Default": "wordpress",
  "Description" : "The WordPress database name",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "64",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBUsername": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account username",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "16",
  "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
  "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},

"DBPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "The WordPress database admin account password",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"DBRootPassword": {
  "Default": "admin",
  "NoEcho": "true",
  "Description" : "Root password for MySQL",
  "Type": "String",
  "MinLength": "1",
  "MaxLength": "41",
  "AllowedPattern" : "[a-zA-Z0-9]*",
  "ConstraintDescription" : "must contain only alphanumeric characters."
},

"LinuxDistribution": {
  "Default": "F17",
  "Description" : "Distribution of choice",
  "Type": "String",
  "AllowedValues" : [ "F18", "F17", "U10", "RHEL-6.1", "RHEL-6.2", "RHEL-6.3" ]
}
  },



 "Mappings" : {
    "AWSInstanceType2Arch" : {
      "m1.tiny" : { "Arch" : "32" },
      "m1.small" : { "Arch" : "64" },
      "m1.medium" : { "Arch" : "64" },
      "m1.large" : { "Arch" : "64" },
      "m1.xlarge" : { "Arch" : "64" }
    },
    "DistroArch2AMI": {
      "F18" : { "32" : "F18-i386-cfntools", "64" : "F18-x86_64-cfntools" },
      "F17" : { "32" : "F17-i386-cfntools", "64" : "F17-x86_64-cfntools" },
      "U10" : { "32" : "U10-i386-cfntools", "64" : "U10-x86_64-cfntools" }
    }
  },

  "Resources" : {
    "CfnUser" : {
      "Type" : "AWS::IAM::User"
    },
    "WebServerKeys" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : {
        "UserName" : {"Ref": "CfnUser"}
      }
    },
    "WebServerGroup" : {
      "Type" : "AWS::AutoScaling::AutoScalingGroup",
      "Properties" : {
        "AvailabilityZones" : { "Fn::GetAZs" : ""},
        "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
        "MinSize" : "1",
        "MaxSize" : "3"
      }
    },
"WebServerScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "1"
  }
},

"WebServerScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebServerGroup" },
    "Cooldown" : "60",
    "ScalingAdjustment" : "-1"
  }
},

 "MEMAlarmHigh": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleUpPolicy" } ],
    "comparison_operator": "gt",
    "description": "high",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "max",
    "threshold": 150
  }
},

"MEMAlarmLow": {
  "Type": "OS::Ceilometer::Alarm",
  "Properties": {
    "alarm_actions": [ { "Ref": "WebServerScaleDownPolicy" } ],
    "comparison_operator": "lt",
    "description": "low",
    "evaluation_periods": 1,
    "meter_name": "testcounter",
    "period": 10,
    "statistic": "min",
    "threshold": 15
  }
},

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Metadata" : {
    "AWS::CloudFormation::Init" : {
      "config" : {
        "files" : {
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys",
                                "SecretAccessKey"]}, "\n"
            ]]},
            "mode" : "000400",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/setup.mysql" : {
            "content" : { "Fn::Join" : ["", [
            "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n",
            "GRANT ALL PRIVILEGES ON ", { "Ref" : "DBName" },
            ".* TO '", { "Ref" : "DBUsername" }, "'@'localhost'\n",
            "IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n",
            "FLUSH PRIVILEGES;\n",
            "EXIT\n"
              ]]},
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/tmp/stats-crontab.txt" : {
            "content" : { "Fn::Join" : ["", [
            "MAIL=\"\"\n",
            "\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmHigh --mem-util\n",
            "* * * * * /opt/aws/bin/cfn-push-stats --watch MEMAlarmLow --mem-util\n"
            ]]},
            "mode" : "000600",
            "owner" : "root",
            "group" : "root"
          }
        },
        "packages" : {
          "yum" : {
            "python-psutil": [],
            "cronie" : [],
            "mysql" : [],
            "mysql-server" : [],
            "httpd" : [],
            "wordpress" : []
          }
        },
        "services" : {
          "systemd" : {
            "mysqld" : { "enabled" : "true", "ensureRunning" : "true" },
            "httpd" : { "enabled" : "true", "ensureRunning" : "true" },
            "crond" : { "enabled" : "true", "ensureRunning" : "true" }
          }
        }
      }
    }
  },
  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "LinuxDistribution" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType" : { "Ref" : "InstanceType" },
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash -v\n",

  "echo \"http_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"https_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"ftp_proxy=<proxy>\" >> /etc/environment\n",
  "echo \"all_proxy=<proxy>\" >> /etc/environment\n",
  "export http_proxy=<proxy>\n",
  "export https_proxy=<proxy>\n",
  "export ftp_proxy=<proxy>\n",
  "export all_proxy=<proxy>\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
      " -r LaunchConfig ",
      " --region ", { "Ref" : "AWS::Region" }, "\n",

      "# Setup MySQL root password and create a user\n",
      "mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'\n",

      "mysql -u root --password='", { "Ref" : "DBRootPassword" },
      "' < /tmp/setup.mysql\n",

      "sed -i \"/Deny from All/d\" /etc/httpd/conf.d/wordpress.conf\n",
      "sed --in-place --e s/database_name_here/", { "Ref" : "DBName" },
      "/ --e s/username_here/", { "Ref" : "DBUsername" },
      "/ --e s/password_here/", { "Ref" : "DBPassword" },
      "/ /usr/share/wordpress/wp-config.php\n",
      "systemctl restart httpd.service\n",

      "# install crontab\n",
      "crontab /tmp/stats-crontab.txt\n"
    ]]}}
  }
}

} }