Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Invalid user token - deferring reject downstream (swift proxy)

My test environment looks like:

 - controller (keystone) - 10.10.1.111
 - swift (proxy) - 10.10.1.112
 - storage(cluster1) - 10.10.1.113
 - storage(cluster2) - 10.10.1.115

I, finally, got everything working (or so I thought). From my proxy server I was able to run the "test" commands and get a result back from keystone: (using http://10.10.1.111:35357/v2.0)

[root@openstack_swift ~(swift)]# swift --debug list
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://controller:35357/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "service", "passwordCredentials": {"username": "swift", "password": "swift"}}}'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): controller
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 1136
DEBUG:keystoneclient.session:RESP: [200] {'date': 'Thu, 19 Jun 2014 03:51:03 GMT', 'content-type': 'application/json', 'content-length': '1136', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-06-19T03:51:03.586599", "expires": "2014-06-19T04:51:03Z", "id": "8a53f47c41e54eb3b807b6dbd806903e", "tenant": {"description": "Service Tenant", "enabled": true, "id": "f1458b9e3c8c4d1388671a322d145799", "name": "service"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.10.1.112:8080/v1", "region": "regionOne", "internalURL": "http://10.10.1.112:8080/v1/AUTH_f1458b9e3c8c4d1388671a322d145799", "id": "72ebd90eb8fb4b629445acd1cbca2152", "publicURL": "http://10.10.1.112:8080/v1/AUTH_f1458b9e3c8c4d1388671a322d145799"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://controller:35357/v2.0", "region": "regionOne", "internalURL": "http://controller:5000/v2.0", "id": "aae73a88b50a4886990521677f494890", "publicURL": "http://controller:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "swift", "roles_links": [], "id": "184c513932514611a0b0d6175a6a9167", "roles": [{"name": "admin"}], "name": "swift"}, "metadata": {"is_admin": 0, "roles": ["d6fef3ca810c485ca1dadc675d102442"]}}}

DEBUG:iso8601.iso8601:Parsed 2014-06-19T04:51:03Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'04', 'daydash': u'19', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'03', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'06', 'day': None, 'minute': u'51'} with default timezone <iso8601.iso8601.Utc object at 0x134da10>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'06' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 6 for 'month' with default 6
DEBUG:iso8601.iso8601:Got u'19' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 19 for 'day' with default 19
DEBUG:iso8601.iso8601:Got u'04' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'51' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'03' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.1.112
DEBUG:urllib3.connectionpool:"GET /v1/AUTH_f1458b9e3c8c4d1388671a322d145799?format=json HTTP/1.1" 200 51
DEBUG:swiftclient:REQ: curl -i http://10.10.1.112:8080/v1/AUTH_f1458b9e3c8c4d1388671a322d145799?format=json -X GET -H "X-Auth-Token: 8a53f47c41e54eb3b807b6dbd806903e"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: [('content-length', '51'), ('accept-ranges', 'bytes'), ('x-timestamp', '1402916327.54569'), ('x-trans-id', 'txc7649b843c8a481cb7d71-0053a25df7'), ('date', 'Thu, 19 Jun 2014 03:50:17 GMT'), ('x-account-bytes-used', '24996082'), ('x-account-container-count', '1'), ('content-type', 'application/json; charset=utf-8'), ('x-account-object-count', '5')]
DEBUG:swiftclient:RESP BODY: [{"count": 5, "bytes": 24996082, "name": "backup"}]
backup
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.1.112
DEBUG:urllib3.connectionpool:"GET /v1/AUTH_f1458b9e3c8c4d1388671a322d145799?format=json&marker=backup HTTP/1.1" 200 2
DEBUG:swiftclient:REQ: curl -i http://10.10.1.112:8080/v1/AUTH_f1458b9e3c8c4d1388671a322d145799?format=json&marker=backup -X GET -H "X-Auth-Token: 8a53f47c41e54eb3b807b6dbd806903e"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: [('content-length', '2'), ('accept-ranges', 'bytes'), ('x-timestamp', '1402916327.54569'), ('x-trans-id', 'txbcd080996485487ab88f5-0053a25dfa'), ('date', 'Thu, 19 Jun 2014 03:50:18 GMT'), ('x-account-bytes-used', '24996082'), ('x-account-container-count', '1'), ('content-type', 'application/json; charset=utf-8'), ('x-account-object-count', '5')]
DEBUG:swiftclient:RESP BODY: []

I can even download and upload files to the storage nodes and I can see that replication is working as expected. My understanding of the proxy node is that you would only place the proxy nodes on the "internet" and that clients would not directly connect to the keystone server.

I found a swift client called cloudberry and when I setup the swift account I, initially, tested it using the keystone information:

User: swift
Pass: swift
Tenant: service
URL: http://10.10.1.111:5000/v2.0

I can then connect and it will list out my container (backup) and once I open that container I am able to see files I have put there through the swift cli and also download/upload additional files.

When I change the swift account URL to http://10.10.1.112:8080/v1 I get an error on the proxy server stating:

Jun 19 11:30:02 openstack_swift proxy-server: Invalid user token - deferring reject downstream
Jun 19 11:31:42 openstack_swift proxy-server: ERROR WSGI: code 400, message Bad request syntax ('{"auth":{"passwordCredentials":{"username":"swift","password":"swift"},"tenantName":"service"}}') (txn: txb866e1ca08cd487db046c-0053a2593a)

From the swift CLI if I use the same user/pass/service with the --debug switch I get the following:

[root@openstack_swift ~(swift)]# swift --debug list
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://10.10.1.112:8080/v1/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "service", "passwordCredentials": {"username": "swift", "password": "swift"}}}'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.1.112
DEBUG:urllib3.connectionpool:"POST /v1/tokens HTTP/1.1" 401 131
DEBUG:keystoneclient.session:RESP: [401] {'date': 'Thu, 19 Jun 2014 03:51:36 GMT', 'content-length': '131', 'content-type': 'text/html; charset=UTF-8', 'www-authenticate': 'Swift realm="tokens"', 'x-trans-id': 'tx510b6e342b3447f982b9b-0053a25e48'}
RESP BODY: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>

DEBUG:keystoneclient.session:Request returned failure status: 401
DEBUG:keystoneclient.v2_0.client:Authorization Failed.
ERROR:swiftclient:Unauthorised. Check username, password and tenant name/id
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1184, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1158, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 345, in get_auth
    insecure=insecure)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 279, in get_keystoneclient_2_0
    raise ClientException('Unauthorised. Check username, password'
ClientException: Unauthorised. Check username, password and tenant name/id
Unauthorised. Check username, password and tenant name/id

Proxy-server.conf

[root@openstack_swift ~(swift)]# cat /etc/swift/proxy-server.conf
[DEFAULT]
bind_port = 8080
workers = 8
user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystone proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:cache]
use = egg:swift#memcache
memcache_servers = 10.10.1.112:11211

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
admin_tenant_name = service
admin_user = swift
admin_password = swift
auth_host = 10.10.1.111
auth_port = 35357
auth_uri = http://10.10.1.111:5000
auth_protocol = http
delay_auth_decision = true
admin_token = 5b8c1336c90880507236
signing_dir = /tmp/keystone-signing-swift
include_service_catalog = false

[filter:keystone]
use = egg:swift#keystoneauth
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
operator_roles = admin, swiftoperator
is_admin = true
cache = swift.cache
keystone_admin_token = 5b8c1336c90880507236

Am I wrong to think that users would connect to the proxy on port 8080 (in production I would expect port 80/443) and then the authentication would then proxy back to the backend authentication server?

I am hoping someone can give me some guidance. I think keystone is not the issue here, but something with the way the proxy is configured. I followed the documentation here to configure all of the servers:

http://docs.openstack.org/icehouse/install-guide/install/yum/content/installing-and-configuring-the-proxy-node.html

Oddly enough things don't "just work" with the default config...