Ask Your Question
0

Getting s3 API to work

asked 2011-04-25 13:46:02 -0500

lgoikhburg gravatar image

Hello,

I'm trying to get Swift3 middleware to work and receive the following exception

glior@fubar:~/s3$ curl -D - -H 'Date: Mon, 25 Apr 2011 17:06:00 +0000' -H 'Authorization: AWS AKIAJIGCCZ3KCSNGLRYQ:YZ+HhhzFyOcFJpi4NdO7THfvVjU=' -L http://10.10.1.29:8080/ HTTP/1.1 500 Internal Server Error Content-Type: text/plain Content-Length: 1087 Date: Mon, 25 Apr 2011 13:43:25 GMT Connection: close

Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/eventlet/wsgi.py", line 336, in handle_one_response result = self.application(self.environ, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/healthcheck.py", line 38, in __call__ return self.app(env, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/memcache.py", line 32, in __call__ return self.app(env, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/swift3.py", line 468, in __call__ res = getattr(controller, req.method)(env, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/swift3.py", line 171, in GET body_iter = self.app(env, self.do_start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/swauth.py", line 133, in __call__ groups = self.get_groups(env, token) File "/usr/lib/pymodules/python2.6/swift/common/middleware/swauth.py", line 203, in get_groups account, user, sign = account.split(':') ValueError: need more than 2 values to unpack

Nothing get's logged in the log. I receive this exception even when Swift3 middleware is disabled in the /etc/swift/proxy-server.conf

[DEFAULT] #cert_file = /etc/swift/cert.crt #key_file = /etc/swift/cert.key bind_port = 8080 workers = 8 user = swift

[pipeline:main] pipeline = healthcheck cache swift3 swauth proxy-server

[app:proxy-server] use = egg:swift#proxy allow_account_management = true

[filter:swift3] use = egg:swift#swift3 log_facility = LOG_LOCAL1

[filter:swauth] use = egg:swift#swauth default_swift_cluster = local#http://10.10.1.29:8080/v1

Highly recommended to change this key to something else!

super_admin_key = swauthkey

[filter:healthcheck] use = egg:swift#healthcheck

[filter:cache] use = egg:swift#memcache memcache_servers = 10.10.1.29:11211

Any idea how to work around this problem ?

edit retag flag offensive close merge delete

18 answers

Sort by ยป oldest newest most voted
0

answered 2012-07-21 16:14:31 -0500

Would you mind sharing how your problem was solved ?

I am facing the same issue now and have no idea where to look for answers.

edit flag offensive delete link more
0

answered 2011-11-08 06:37:29 -0500

akapadia-usa gravatar image

Please ignore my post. Problem Solved. Thanks.

edit flag offensive delete link more
0

answered 2011-11-06 00:36:59 -0500

akapadia-usa gravatar image

Also, I'm attaching logs results from both the failed request (s3curl) and successful request (python boto):

Log results from s3curl (failure)

Nov 6 00:23:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/23/21 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0145 Nov 6 00:23:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/23/21 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0047

Log results from boto library (success)

Bucket listing

Nov 6 00:28:01 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/01 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0055 Nov 6 00:28:01 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/01 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0097 Nov 6 00:28:01 ip-184-73-4-50 proxy-server 99.41.51.203 99.41.51.203 06/Nov/2011/00/28/01 GET /v1/AUTH_68d78d77-8977-44d3-94eb-b9f60769e8fb/--help%3Fformat%3Djson%26limit%3D1 HTTP/1.0 404 - Boto/2.1.0%20%28linux2%29 test%3Atester%2Cs3 - - - tx9ee5348c12df46a994d20c15dc97ae73 - 0.0464 Nov 6 00:28:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/21 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0047 Nov 6 00:28:21 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/21 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0045 Nov 6 00:28:21 ip-184-73-4-50 proxy-server 99.41.51.203 99.41.51.203 06/Nov/2011/00/28/21 GET /v1/AUTH_68d78d77-8977-44d3-94eb-b9f60769e8fb/--help%3Fformat%3Djson%26limit%3D1 HTTP/1.0 404 - Boto/2.1.0%20%28linux2%29 test%3Atester%2Cs3 - - - tx7cea5f0934d0431680eef12b02986eaf - 0.0094 Nov 6 00:28:44 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/44 GET /v1/AUTH_.auth/test/tester HTTP/1.0 200 - Swauth - - - - - - 0.0045 Nov 6 00:28:44 ip-184-73-4-50 proxy-server - - 06/Nov/2011/00/28/44 HEAD /v1/AUTH_.auth/test HTTP/1.0 204 - Swauth - - - - - - 0.0049 Nov 6 00:28:45 ip-184-73-4-50 proxy-server 99.41.51.203 99.41.51.203 06/Nov/2011/00/28/45 GET /v1/AUTH_68d78d77-8977-44d3-94eb-b9f60769e8fb%3Fformat%3Djson HTTP/1.0 200 - Boto/2.1.0%20%28linux2%29 test%3Atester%2Cs3 - 122 - txb159ba00f4274a9da944b5b8bf8348fc - 0.0055

edit flag offensive delete link more
0

answered 2011-11-05 20:27:35 -0500

akapadia-usa gravatar image

Did this problem get solved? I changed the s3curl.pl file as pointed out, but I'm still having problems. Help appreciated.

Thanks, Amar

akapadia@ubuntu:~/s3-curl$ ./s3curl.pl --id 'test:tester' --key 'testing' --get -- -s -v -k https://ec2-184-73-4-50.compute-1.amazonaws.com (https://ec2-184-73-4-50.compute-1.ama...) Unknown option: get WARNING: It isn't safe to put your AWS secret access key on the command line! The recommended key management system is to store your AWS secret access keys in a file owned by, and only readable by you.

For example:

%awsSecretAccessKeys = ( # personal account personal => { id => '1ME55KNV6SBTR7EXG0R2', key => 'zyMrlZUKeG9UcYpwzlPko/+Ciu0K2co0duRM3fhi', },

# corporate account
company => {
    id => '1ATXQ3HHA59CYF1CVS02',
    key => 'WQY4SrSS95pJUT95V6zWea01gBKBCL6PI0cdxeH8',
},

);

$ chmod 600 /home/akapadia/.s3curl

Will sleep and continue despite this problem. Please set up /home/akapadia/.s3curl for future requests. * About to connect() to http://ec2-184-73-4-50.compute-1.amazonaws.com port 443 (#0) * Trying 184.73.4.50... connected * Connected to http://ec2-184-73-4-50.compute-1.amazonaws.com (184.73.4.50) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-SHA * Server certificate: * subject: C=US; ST=CA; L=San Jose; O=OpenStack Consulting; OU=Swift; http://CN=184-73-4-50.compute-1.amazonaws.com ; emailAddress=bitnami@ec2-184-73-4-50.... * start date: 2011-11-04 04:26:58 GMT * expire date: 2011-12-04 04:26:58 GMT * common name: http://ec2-184-73-4-50.compute-1.amazonaws.com (matched) * issuer: C=US; ST=CA; L=San Jose; O=OpenStack Consulting; OU=Swift; http://CN=ec2-184-73-4-50.compute-1.amazonaws.com (CN=ec2-184-73-4-50.compute-1.amazonaw...) ; emailAddress=bitnami@ec2-184-73-4-50.... * SSL certificate verify result: self signed certificate (18), continuing anyway.

GET / HTTP/1.1 User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 Host: http://ec2-184-73-4-50.compute-1.amazonaws.com Accept: / Date: Sat, 05 Nov 2011 20:21:03 +0000 Authorization: AWS test:tester:FakJ/xP9Xv16g77QR4S/8CRpoCI=

< HTTP/1.1 403 Forbidden < Content-Type: text/xml; charset=UTF-8 < Content-Length: 124 < Date: Sat, 05 Nov 2011 20:21:04 GMT < <error> AccessDeniedAccess denied * Connection #0 to host http://ec2-184-73-4-50.compute-1.amazonaws.com left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1):

edit flag offensive delete link more
0

answered 2011-05-05 21:16:43 -0500

cthier gravatar image

Resolution was discussed on IRC, but adding here for completeness sake. If you want to use s3curl.pl, you will have to modify the script and add your storage hostname to @endpoints. For example, I made this work on my dev instance by adding 'localhost' to the list.

edit flag offensive delete link more
0

answered 2011-04-27 16:22:42 -0500

cthier gravatar image

I downloaded s3curl.pl to give it a try.

While it did seem to authorize correctly, I got a 403, and when I looked closer I noticed that s3curl.pl wasn't signing the request correctly. My guess is that maybe s3curl.pl is having problems with the : in the key? I'm not familiar enough with perl thought to try to debug it.

Most of our testing has been done using the boto library.

edit flag offensive delete link more
0

answered 2011-04-26 23:31:51 -0500

jinzishuai gravatar image

OK, that's back to my original post.

From the auth server, I obtained the following: seki@OS-CC:/var/log$ curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.33:11000/v1.0 ... < X-Storage-Url: https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197 (https://192.168.1.33:8080/v1/AUTH_365...) < X-Storage-Token: AUTH_tka3599de5039545809d181637d0f010a9 < X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9 I guess my URL would be https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197 (https://192.168.1.33:8080/v1/AUTH_365...)

So I am trying to access it as

./s3curl.pl --id system:root --key testpass --get -- -s -v -k https://192.168.1.33:8080/AUTH_365f77c9d523435dbcf12c9d2678d197 (https://192.168.1.33:8080/AUTH_365f77...) Still got 401 error.

Note the following works perfectly well but it is not S3 compatible.

seki@OS-CC:~/s3-curl$ curl -k -v -H 'X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9' https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197 (https://192.168.1.33:8080/v1/AUTH_365...) * About to connect() to 192.168.1.33 port 8080 (#0) * Trying 192.168.1.33... connected * Connected to 192.168.1.33 (192.168.1.33) port 8080 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-SHA * Server certificate: * subject: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; emailAddress=Shi.Jin@vrstorm.com * start date: 2011-04-23 15:55:37 GMT * expire date: 2011-05-23 15:55:37 GMT * common name: OS-CC (does not match '192.168.1.33') * issuer: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; emailAddress=Shi.Jin@vrstorm.com * SSL certificate verify result: self signed certificate (18), continuing anyway.

GET /v1/AUTH_365f77c9d523435dbcf12c9d2678d197 HTTP/1.1 User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18 Host: 192.168.1.33:8080 Accept: / X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9

< HTTP/1.1 200 OK < X-Account-Object-Count: 1 < X-Account-Bytes-Used: 9172 < X-Account-Container-Count: 1 < Content-Length: 13 < Content-Type: text/plain; charset=utf8 < Date: Tue, 26 Apr 2011 23:28:41 GMT < newcontainer * Connection #0 to host 192.168.1.33 left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1):

Thanks.

edit flag offensive delete link more
0

answered 2011-04-26 23:06:49 -0500

cthier gravatar image

You url should look something like http://localhost:8080/v1.0/ACCOUNTHASH/CONTAINER/OBJECT (http://localhost:8080/v1.0/ACCOUNTHAS...)

edit flag offensive delete link more
0

answered 2011-04-26 22:54:15 -0500

jinzishuai gravatar image

Thank you but I am still in error.

seki@OS-CC:~/s3-curl$ ./s3curl.pl --id system:root --key testpass --get -- -s -v https://localhost:8080/ -k ... * About to connect() to localhost port 8080 (#0) * Trying ::1... Connection refused * Trying 127.0.0.1... connected * Connected to localhost (127.0.0.1) port 8080 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-SHA * Server certificate: * subject: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; emailAddress=Shi.Jin@vrstorm.com * start date: 2011-04-23 15:55:37 GMT * expire date: 2011-05-23 15:55:37 GMT * common name: OS-CC (does not match 'localhost') * issuer: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; emailAddress=Shi.Jin@vrstorm.com * SSL certificate verify result: self signed certificate (18), continuing anyway.

GET / HTTP/1.1 User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18 Host: localhost:8080 Accept: / Date: Tue, 26 Apr 2011 22:51:07 +0000 Authorization: AWS system:root:y/MiNg2xGvRuvLJ4qL5FSTZEjGw=

< HTTP/1.1 401 Unauthorized < Content-Type: text/html; charset=UTF-8 < Content-Length: 364 < Date: Tue, 26 Apr 2011 22:51:07 GMT < <html> <head> <title>401 Unauthorized</title> </head> <body>

401 Unauthorized

This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.

</body> * Connection #0 to host localhost left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1):

I also tried to connect to https://localhost:8080/v1 but got the same 401 error.

Could you please help? Thanks. Shi

edit flag offensive delete link more
0

answered 2011-04-26 22:37:38 -0500

cthier gravatar image

Yes the blueprint isn't correct anymore. That was the first approach that I used in my proof of concept.

You should connect to the proxy just as you would when normally accessing the data, and since you are signing your request, you don't have to make a call to auth to get a token.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-04-25 13:46:02 -0500

Seen: 653 times

Last updated: Jul 21 '12