Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Authentication failed when setting up two region openstack deployment using devstack

Dear Stackers,

We are trying to use devstack to deploy two OpenStack in different regions with shared keystone service which is installed in region one(dc1). The first region(dc1) deployment was successful, however when we deployed the second OpenStack in second region(dc2), authentication failed when uploading the image via glance,even more the keystone in first region is corrupted with error code 400.

Note : This issue can also be reproduced in juno release.

Any ideas are appreciated!

Here are the details configuration. local file in region one

[[local|localrc]]
REGION_NAME=dc1
OFFLINE=False
GIT_BASE=https://github.com
HOST_IP=192.85.180.138
LOGDAYS=1
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
VERBOSE=TRUE
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
RECLONE=yes
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
IMAGE_URLS+=",http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2"
IMAGE_URLS+=",http://cloud.fedoraproject.org/fedora-20.x86_64.qcow2"
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_LOOPBACK_DISK_SIZE=4G
ENABLED_SERVICES+=,horizon
ENABLED_SERVICES+=,sahara
KEYSTONE_TOKEN_FORMAT=PKI
Q_USE_DEBUG_COMMAND=True
Q_USE_SECGROUP=True
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector
enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier
enable_service ceilometer-api
CEILOMETER_BACKEND=mongo
enable_service heat h-api h-api-cfn h-api-cw h-eng

local.conf file in region two(dc2)

[[local|localrc]]
KEYSTONE_SERVICE_HOST=192.85.180.138
KEYSTONE_AUTH_HOST=192.85.180.138
OFFLINE=False
GIT_BASE=https://github.com
HOST_IP=192.85.180.140
LOGDAYS=1
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
VERBOSE=TRUE
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
RECLONE=yes
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
IMAGE_URLS+=",http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2"
IMAGE_URLS+=",http://cloud.fedoraproject.org/fedora-20.x86_64.qcow2"
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_LOOPBACK_DISK_SIZE=4G
ENABLED_SERVICES+=,sahara
KEYSTONE_TOKEN_FORMAT=PKI
Q_USE_DEBUG_COMMAND=True
Q_USE_SECGROUP=True
enable_service heat h-api h-api-cfn h-api-cw h-eng
REGION_NAME=dc2
disable_service horizon

The error messages in dc2 is

2014-10-20 21:22:45.812 | ERROR: openstack <html>
2014-10-20 21:22:45.812 |  <head>
2014-10-20 21:22:45.812 |   <title>401 Unauthorized</title>
2014-10-20 21:22:45.812 |  </head>
2014-10-20 21:22:45.812 |  <body>
2014-10-20 21:22:45.812 |   <h1>401 Unauthorized</h1>
2014-10-20 21:22:45.812 |   This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br /><br />

The error message in dc1 is

stack@sahara:~/devstack$ keystone --d user-list
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.85.180.138:5000/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.85.180.138
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 17360
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.85.180.138:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: {SHA1}52faef8c871f049b9278debceae02babca52dcd6"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.85.180.138
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"GET /v2.0/users HTTP/1.1" 400 388
DEBUG:keystoneclient.session:RESP:
DEBUG:keystoneclient.session:Request returned failure status: 400
Bad Request (HTTP 400)

keystone server error log

2014-10-20 17:26:52.573578 25740 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /opt/stack/keystone/keystone/middleware/core.py:270
2014-10-20 17:26:52.577577 25740 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /opt/stack/keystone/keystone/common/wsgi.py:191

Authentication failed when setting up two region openstack deployment using devstack

Dear Stackers,

We are trying to use devstack to deploy two OpenStack in different regions with shared keystone service which is installed in region one(dc1). The first region(dc1) deployment was successful, however when we deployed the second OpenStack in second region(dc2), authentication failed when uploading the image via glance,even more the keystone in first region is corrupted with error code 400.

UPDATE: if we changed the token format to UUID, glance can upload image with following error message and the horizon dashboard can be opened, however we prefer to use PKI cos' it is more secure, any hint?

...
    85.175.142 - - [22/Oct/2014 18:04:46] "POST /v1/images HTTP/1.1" 201 831 370.017314
    16.85.175.142 - - [22/Oct/2014 18:04:46] code 400, message Bad request syntax ('0')
    16.85.175.142 - - [22/Oct/2014 18:04:46] "0" 400 -
...

Note : This issue can also be reproduced in juno release.

Any ideas are appreciated!

Here are the details configuration. local file in region one

[[local|localrc]]
REGION_NAME=dc1
OFFLINE=False
GIT_BASE=https://github.com
HOST_IP=192.85.180.138
LOGDAYS=1
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
VERBOSE=TRUE
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
RECLONE=yes
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
IMAGE_URLS+=",http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2"
IMAGE_URLS+=",http://cloud.fedoraproject.org/fedora-20.x86_64.qcow2"
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_LOOPBACK_DISK_SIZE=4G
ENABLED_SERVICES+=,horizon
ENABLED_SERVICES+=,sahara
KEYSTONE_TOKEN_FORMAT=PKI
Q_USE_DEBUG_COMMAND=True
Q_USE_SECGROUP=True
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector
enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier
enable_service ceilometer-api
CEILOMETER_BACKEND=mongo
enable_service heat h-api h-api-cfn h-api-cw h-eng

local.conf file in region two(dc2)

[[local|localrc]]
KEYSTONE_SERVICE_HOST=192.85.180.138
KEYSTONE_AUTH_HOST=192.85.180.138
OFFLINE=False
GIT_BASE=https://github.com
HOST_IP=192.85.180.140
LOGDAYS=1
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
VERBOSE=TRUE
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password
RECLONE=yes
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
IMAGE_URLS+=",http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2"
IMAGE_URLS+=",http://cloud.fedoraproject.org/fedora-20.x86_64.qcow2"
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_LOOPBACK_DISK_SIZE=4G
ENABLED_SERVICES+=,sahara
KEYSTONE_TOKEN_FORMAT=PKI
Q_USE_DEBUG_COMMAND=True
Q_USE_SECGROUP=True
enable_service heat h-api h-api-cfn h-api-cw h-eng
REGION_NAME=dc2
disable_service horizon

The error messages in dc2 is

2014-10-20 21:22:45.812 | ERROR: openstack <html>
2014-10-20 21:22:45.812 |  <head>
2014-10-20 21:22:45.812 |   <title>401 Unauthorized</title>
2014-10-20 21:22:45.812 |  </head>
2014-10-20 21:22:45.812 |  <body>
2014-10-20 21:22:45.812 |   <h1>401 Unauthorized</h1>
2014-10-20 21:22:45.812 |   This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br /><br />

The error message in dc1 is

stack@sahara:~/devstack$ keystone --d user-list
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.85.180.138:5000/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.85.180.138
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 17360
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.85.180.138:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: {SHA1}52faef8c871f049b9278debceae02babca52dcd6"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.85.180.138
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"GET /v2.0/users HTTP/1.1" 400 388
DEBUG:keystoneclient.session:RESP:
DEBUG:keystoneclient.session:Request returned failure status: 400
Bad Request (HTTP 400)

keystone server error log

2014-10-20 17:26:52.573578 25740 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /opt/stack/keystone/keystone/middleware/core.py:270
2014-10-20 17:26:52.577577 25740 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /opt/stack/keystone/keystone/common/wsgi.py:191