Ask Your Question
0

Heat Autoscaling Webhook authorization failure

asked 2014-11-10 03:59:41 -0500

mc__ gravatar image

updated 2014-11-10 04:24:49 -0500

Hi there,

At the moment I'm testing the autoscaling feature of Openstack Icehouse. Therefore I used an http://paste.openstack.org/show/131369/ (self-modified) version of a predefined HOT from https://github.com/openstack/heat-templates/blob/master/hot/autoscaling.yaml (OpenStack @ github)
Openstack itself was initialized and configurated with a given puppet script by my division.

When I stress my VM in the web-server-group to test the triggering of another VM, no adjustment happens, although Ceilometers alarm-history shows successful breaches.

stress command: '*stress --cpu 8 --io 8 --vm 2 --vm-bytes 256M &*' # causing permanent usage of ~60% CPU usage.

user@foo: ceilometer alarm-history -a 69edb170-c3d2-4e3e-98bb-40b99e266a4e
+------------------+----------------------------+-------------------------------------------------------------+
| Type             | Timestamp                  | Detail                                                      |
+------------------+----------------------------+-------------------------------------------------------------+
| creation         | 2014-11-06T02:47:21.514014 | name: TestStack-cpu_alarm_high-jft5kuz42vzx                 |
|                  |                            | description: Scale-up if the average CPU > 50% for 1 minute |
|                  |                            | type: threshold                                             |
|                  |                            | rule: cpu_util > 50.0 during 1 x 60s                        |
| state transition | 2014-11-06T03:04:03.772689 | state: ok                                                   |
| state transition | 2014-11-06T03:06:03.769424 | state: insufficient data                                    |
| state transition | 2014-11-06T03:14:03.844720 | state: ok                                                   |
| state transition | 2014-11-06T03:16:03.834233 | state: insufficient data                                    |
| state transition | 2014-11-06T03:24:07.262171 | state: alarm                                                |
| state transition | 2014-11-06T03:26:03.895120 | state: insufficient data                                    |
| state transition | 2014-11-06T03:34:03.979063 | state: alarm                                                |
| state transition | 2014-11-06T03:36:03.963650 | state: insufficient data                                    |
+------------------+----------------------------+-------------------------------------------------------------+

Triggering the webhook manually leads into an error-message:

curl -XPOST -i "http://xxx.xxx.xxx.xxx:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3A9036a06bb9f648beb8b4d4592e693735%3Astacks%2FTestStack%2F3d0054f4-401a-4578-b9fd-4b6374bc69b2%2Fresources%2Fweb_server_scaleup_policy?Timestamp=2014-11-06T02%3A47%3A19Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=e4f6cefc18a34695b1aba1bd5e0bbb0a&SignatureVersion=2&Signature=dOjsDdfO37Ym0%2F3t5A1qzxNmrjfYHdohJZ%2FQ3HBJafU%3D"

user@foo:  HTTP/1.1 403 AccessDenied
Content-Type: application/xml; charset=UTF-8
Content-Length: 149
Date: Thu, 06 Nov 2014 09:30:00 GMT

<ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse>

The keystone-all.log shows following messages related directly to the webhook action:

2014-11-06 18:17:43.194 26800 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:271
2014-11-06 18:17:43.197 26800 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.xxx- - [06/Nov/2014 18:17:43] "POST /v2.0/ec2tokens/ec2tokens HTTP/1.1" 404 252 0.003501

And heat-engine.log contains that:

2014-11-06 18:50:22.679 28354 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::9036a06bb9f648beb8b4d4592e693735:stacks/TestStack/3d0054f4-401a-4578-b9fd-4b6374bc69b2/resources/web_server_scaleup_policy Accept: */* process_request /usr/lib/python2.7/dist-packages/heat/api/middleware/version_negotiation.py:53
2014-11-06 18:50:22.680 28354 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /usr/lib/python2.7/dist-packages/heat/api/middleware/version_negotiation.py:68
2014-11-06 18:50:22.680 28354 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-11-06 18:50:22.681 28354 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-11-06 18:50:22.682 28354 INFO heat.api.aws.ec2token [-] Authenticating with http://xxx.xxx.xxx.xxx:5000/v2.0/ec2tokens/ec2tokens
2014-11-06 18:50:22.684 28354 INFO urllib3 ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
1

answered 2014-11-10 05:03:55 -0500

shardy gravatar image

I believe this is the same issue as https://bugs.launchpad.net/heat/+bug/... , where the reporter had a spurious ec2tokens in their auth_uri in heat.conf.

We can see the evidence in your engine.log:

http://xxx.xxx.xxx.xxx:5000/v2.0/ec2t...

Changing the ec2tokens auth_uri in your heat.conf to remove the ec2tokens path suffix should resolve the issue.

Note we added logic to tolerate this configuration to Juno heat, perhaps this should be backported to Icehouse, and definitely if there are still puppet manifiests and/or docs which configure heat this way, they should be fixed.

Here's the heat patch:

https://review.openstack.org/#/c/98827/

So, to clarify, change: auth_uri=http://xxx.xxx.xxx.xxx:5000/v2.0/ec2tokens to: auth_uri=http://xxx.xxx.xxx.xxx:5000/v2.0

and it should work fine.

edit flag offensive delete link more
0

answered 2014-11-13 20:20:36 -0500

mc__ gravatar image

updated 2014-11-13 20:22:02 -0500

I used a github puppet script for setup. Can you see any misconfiguration in there? Maybe one of these lines are later extended to ec2tokens/ec2tokens

user@foo:/usr/share/puppet/modules$ grep -r ec2tokens *

heat/spec/classes/heat_init_spec.rb:      :keystone_ec2_uri      => 'http://127.0.0.1:5000/v2.0/ec2tokens',
heat/spec/classes/heat_init_spec.rb:        :keystone_ec2_uri => 'http://1.2.3.4:35357/v2.0/ec2tokens'
heat/spec/classes/heat_init_spec.rb:      should contain_heat_config('ec2authtoken/auth_uri').with_value('http://1.2.3.4:35357/v2.0/ec2tokens')
heat/manifests/api-cfn.pp:  $keystone_ec2_uri  = 'http://127.0.0.1:5000/v2.0/ec2tokens',
heat/manifests/api-cloudwatch.pp:  $keystone_ec2_uri  = 'http://127.0.0.1:5000/v2.0/ec2tokens',
heat/manifests/init.pp:  $keystone_ec2_uri            = 'http://127.0.0.1:5000/v2.0/ec2tokens',
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-11-10 03:59:41 -0500

Seen: 1,434 times

Last updated: Nov 13 '14