Ask Your Question
0

Could not get secret on group scaling

asked 2013-07-31 16:40:37 -0500

davide-m gravatar image

When heat tries to scale a group after an alarm, it is unable to launch the new instance. I think that it can't get the right credentials and uses a null password.

engine.log reports: 2013-07-31 17:14:33.956 4279 DEBUG heat.engine.service [-] Periodic watcher task for stack 8b1a4382-a09f-473f-b197-c41531694484 _periodic_watcher_task /usr/lib/python2.6/site-packages/heat/engine/service.py:511 2013-07-31 17:14:33.966 4279 INFO heat.engine.watchrule [-] WATCH: stack:8b1a4382-a09f-473f-b197-c41531694484, watch_name:wp-single-HA-test261.MEMAlarmHigh ALARM 2013-07-31 17:14:34.017 4279 INFO heat.engine.watchrule [-] WATCH: stack:8b1a4382-a09f-473f-b197-c41531694484, watch_name:wp-single-HA-test261.MEMAlarmLow NORMAL 2013-07-31 17:14:34.017 4279 INFO heat.engine.watchrule [-] no action for new state NORMAL 2013-07-31 17:14:34.023 4279 INFO heat.engine.resources.autoscaling [-] WebServerScaleUpPolicy Alarm, adjusting Group WebServerGroup by 1 2013-07-31 17:14:34.026 4279 DEBUG heat.engine.resources.autoscaling [-] adjusting capacity from 1 to 2 resize /usr/lib/python2.6/site-packages/heat/engine/resources/autoscaling.py:182 2013-07-31 17:14:34.033 4279 INFO heat.engine.resources.autoscaling [-] creating inst 2013-07-31 17:14:34.033 4279 INFO heat.engine.resource [-] creating GroupedInstance "WebServerGroup-1" 2013-07-31 17:14:34.063 4279 WARNING heat.engine.resources.user [-] could not get secret for wp-single-HA-test261.CfnUser Error:Authorization Failed: Unable to communicate with identity service: {"error": {"message": "crypt() argument 1 must be string without null bytes, not str", "code": 400, "title": "Bad Request"}}. (HTTP 400) 2013-07-31 17:14:34.063 4279 INFO heat.engine.resources.user [-] wp-single-HA-test261.WebServerKeys.GetAtt(SecretAccessKey) == <sanitized> 2013-07-31 17:14:34.096 4279 ERROR heat.engine.resource [-] create GroupedInstance "WebServerGroup-1" 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource Traceback (most recent call last): 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 320, in create 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource self.handle_create() 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resources/instance.py", line 253, in handle_create 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource keypairs = [k.name for k in self.nova().keypairs.list()] 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 286, in nova 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource return self.stack.clients.nova(service_type) 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/clients.py", line 104, in nova 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource client.authenticate() 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/v1_1/client.py", line 169, in authenticate 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource self.client.authenticate() 2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 330, in authenticate 2013-07-31 17:14:34.096 4279 ... (more)

edit retag flag offensive close merge delete

8 answers

Sort by ยป oldest newest most voted
0

answered 2013-08-01 16:36:13 -0500

davide-m gravatar image

Reading carefully the keystone log I realize that the username\password that heat provides are right: "4\u0000\u0000\u00002\u0000\u0000\u00009\u0000\u0000\u0000f\u0000\u0000\u00003\u0000\u0000\u00002\u0000\u0000\u0000b\u0000\u0000\u00008\u0000\u0000\u00006\u0000\u0000\u00008\u0000\u0000\u0000a\u0000\u0000\u0000d\u0000\u0000\u00004\u0000\u0000\u00002\u0000\u0000\u00009\u0000\u0000\u0000a\u0000\u0000\u0000" is exactly my admin password, but with three unicode null-char between each character.

Maybe the problem is not heat?

This is the keystone log:

2013-07-30 13:54:51 ERROR [root] crypt() argument 1 must be string without null bytes, not str Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 236, in __call__ result = method(context, *params) File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 79, in authenticate context, auth) File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 301, in _authenticate_local tenant_id=tenant_id) File "/usr/lib/python2.6/site-packages/keystone/common/manager.py", line 47, in _wrapper return f(args, **kw) File "/usr/lib/python2.6/site-packages/keystone/identity/backends/sql.py", line 189, in authenticate if not self._check_password(password, user_ref): File "/usr/lib/python2.6/site-packages/keystone/identity/backends/sql.py", line 170, in _check_password return utils.check_password(password, user_ref.get('password')) File "/usr/lib/python2.6/site-packages/keystone/common/utils.py", line 135, in check_password return passlib.hash.sha512_crypt.verify(password_utf8, hashed) File "/usr/lib/python2.6/site-packages/passlib/utils/handlers.py", line 465, in verify return self.checksum == self.calc_checksum(secret) File "/usr/lib/python2.6/site-packages/passlib/handlers/sha2_crypt.py", line 500, in _calc_checksum_os_crypt ok, result = safe_os_crypt(secret, self.to_string(native=False)) File "/usr/lib/python2.6/site-packages/passlib/utils/__init__.py", line 197, in safe_os_crypt return True, os_crypt(secret, hash).decode("ascii") File "/usr/lib64/python2.6/crypt.py", line 55, in crypt return _crypt.crypt(word, salt) TypeError: crypt() argument 1 must be string without null bytes, not str

Maybe I have wrong version of any library?

Thanks!

edit flag offensive delete link more
0

answered 2013-08-01 01:31:11 -0500

2 questions: - what version of heat are you running? - did you specify a username and password when you launched the stack? (not just an auth_token?)

edit flag offensive delete link more
0

answered 2013-08-01 08:55:42 -0500

davide-m gravatar image

I'm running heat 2013.1.2-3 from grizzly rdo repository.

I'm using the administration credentials from my system environment.

My environment: OS_AUTH_URL="http://CONTROLLER_HOST:35357/v2.0/" OS_PASSWORD="ADMIN_PASSWD" OS_TENANT_NAME="admin" OS_USERNAME="admin"

Client command:

$ heat stack-create -f AutoScalingGroupTest.tempalte -P "DBUsername=user;DBPassword=pass;KeyName=mykey;" test-stack

This is my context debug (if it could be useful):

2013-08-01 10:40:02.488 4279 DEBUG heat.openstack.common.rpc.amqp [-] unpacked context: {'username': None, 'service_user': u'heat', 'service_tenant': u'services', 'roles': [u'_member_', u'heat_stack_user'], 'aws_auth_uri': u'http://CONTROLLER_HOST:5000/v2.0/ec2tokens', 'tenant_id': u'ADMIN_TENANT_ID', 'auth_token': '<sanitized>', 'service_password': u'HEAT_PASS', 'auth_url': u'http://CONTROLLER_HOST:5000/v2.0', 'is_admin': True, 'password': None, 'aws_creds': u'{"ec2Credentials": {"access": "EC2_ACCESS", "signature": "EC2_SIGNATURE"}}', 'tenant': u'admin'} _safe_log /usr/lib/python2.6/site-packages/heat/openstack/common/rpc/common.py:276

Is it right to use administration credentials (admin user on admin tenant) to launch the stack?

edit flag offensive delete link more
0

answered 2013-08-02 09:00:07 -0500

davide-m gravatar image

I did a little and quick hack to fix this issue.

I modified the code of heat/db/sqlalchemy/api.py at line 221, stripping out the null-chars

from

result['password'] = crypt.decrypt(result['password'])

to

result['password'] = crypt.decrypt(result['password']).replace("\x00", "")

Now seems to work fine. But why I have this issue? Is it heat bug?

edit flag offensive delete link more
0

answered 2013-08-02 09:08:44 -0500

zaneb gravatar image

This is clearly an encoding problem - we're getting unicode out but treating it as an ascii/utf-8 string. The answer is definitely not just stripping out the null characters, the correct fix would be to make sure the string is encoded as utf-8 _before_ it is encrypted.

edit flag offensive delete link more
0

answered 2013-08-02 11:09:04 -0500

davide-m gravatar image

Yes, it works as well modifying user_creds_create() in heat/db/sqlalchemy/api.py at line 209

from

user_creds_ref.password = crypt.encrypt(values['password'])

to

user_creds_ref.password = crypt.encrypt(values['password'].encode('utf-8'))

Should I create a bug report?

edit flag offensive delete link more
0

answered 2013-09-12 19:02:40 -0500

I'm facing a similar problem with my password.

I tried davide suggestion but even though my password still contains the \x00 characters.

After browsing this problem around, I found out we may all be facing a unicode conversion problem caused probably by the crypt module.

http://stackoverflow.com/questions/10253405/python-x00-filled-utf-32-string-from-cstringio (http://stackoverflow.com/questions/10...)

edit flag offensive delete link more
0

answered 2013-09-12 19:40:37 -0500

davide-m gravatar image

Try replacing that line with

user_creds_ref.password = crypt.encrypt(unicode(values['password']).encode('utf-8'))

and then restart all heat services.

This workaround for me is working well.

Here there is a proposed patch: https://review.openstack.org/#/c/46287/1/heat/db/sqlalchemy/api.py (https://review.openstack.org/#/c/4628...)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-07-31 16:40:37 -0500

Seen: 107 times

Last updated: Sep 12 '13