Authentication failed when setting up two region openstack deployment using devstack [closed]
Dear Stackers,
We are trying to use devstack to deploy two OpenStack in different regions with shared keystone service which is installed in region one(dc1). The first region(dc1) deployment was successful, however when we deployed the second OpenStack in second region(dc2), authentication failed when uploading the image via glance,even more the keystone in first region is corrupted with error code 400.
UPDATE: if we changed the token format to UUID, glance can upload image with following error message and the horizon dashboard can be opened, however we prefer to use PKI cos' it is more secure, any hint?
...
85.175.142 - - [22/Oct/2014 18:04:46] "POST /v1/images HTTP/1.1" 201 831 370.017314
16.85.175.142 - - [22/Oct/2014 18:04:46] code 400, message Bad request syntax ('0')
16.85.175.142 - - [22/Oct/2014 18:04:46] "0" 400 -
...
Note : This issue can also be reproduced in juno release.
Any ideas are appreciated!
Here are the details configuration. local file in region one
[[local|localrc]]
REGION_NAME=dc1
OFFLINE=False
GIT_BASE=https://github.com
HOST_IP=192.85.180.138
LOGDAYS=1
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
VERBOSE=TRUE
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
RECLONE=yes
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
IMAGE_URLS+=",http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2"
IMAGE_URLS+=",http://cloud.fedoraproject.org/fedora-20.x86_64.qcow2"
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_LOOPBACK_DISK_SIZE=4G
ENABLED_SERVICES+=,horizon
ENABLED_SERVICES+=,sahara
KEYSTONE_TOKEN_FORMAT=PKI
Q_USE_DEBUG_COMMAND=True
Q_USE_SECGROUP=True
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector
enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier
enable_service ceilometer-api
CEILOMETER_BACKEND=mongo
enable_service heat h-api h-api-cfn h-api-cw h-eng
local.conf file in region two(dc2)
[[local|localrc]]
KEYSTONE_SERVICE_HOST=192.85.180.138
KEYSTONE_AUTH_HOST=192.85.180.138
OFFLINE=False
GIT_BASE=https://github.com
HOST_IP=192.85.180.140
LOGDAYS=1
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
VERBOSE=TRUE
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
RECLONE=yes
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
IMAGE_URLS+=",http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2"
IMAGE_URLS+=",http://cloud.fedoraproject.org/fedora-20.x86_64.qcow2"
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_LOOPBACK_DISK_SIZE=4G
ENABLED_SERVICES+=,sahara
KEYSTONE_TOKEN_FORMAT=PKI
Q_USE_DEBUG_COMMAND=True
Q_USE_SECGROUP=True
enable_service heat h-api h-api-cfn h-api-cw h-eng
REGION_NAME=dc2
disable_service horizon
The error messages in dc2 is
2014-10-20 21:22:45.812 | ERROR: openstack <html>
2014-10-20 21:22:45.812 | <head>
2014-10-20 21:22:45.812 | <title>401 Unauthorized</title>
2014-10-20 21:22:45.812 | </head>
2014-10-20 21:22:45.812 | <body>
2014-10-20 21:22:45.812 | <h1>401 Unauthorized</h1>
2014-10-20 21:22:45 ...
Just curious. Why do you think PKI is more secure? Check the signing_dir configured for services. Is the content of signing dir same for keystone and the services? If you didn't configure then most probably it will default to /tmp/
Thanks for the reply,IMHO PKI type token will be much longer and complex which is not easy to be compromised, besides it remove the burden of keystone server to verify the token in a offline mode using public key.The signing dirs are configured by devstack by default it is in /var/cache/svc_name.
Remove the contents of /var/cache/svc_name, restart the service and try it. Most probably you have incompatible certificates from 2 installations. Keystone middleware will fetch proper certificate if that directory is empty
Sorry for the late reply, i have removed the certificates in /var/cache for glance, nova, etc. however i don't know how to restart all the services in devstack, is there any script to restart all the devstack services? Thanks!
There is a script called "rejoin-stack.sh", execute this it will bring up all the services in screen.