Ask Your Question
0

S3 API (s3curl) returns empty content

asked 2016-10-24 16:28:25 -0500

doka.ua gravatar image

updated 2016-10-25 07:07:15 -0500

Hi colleagues,

I'm trying to build S3 test connection for our developers using devstack. I configured only horizon, keystone and swift (using local.conf):

disable_all_services
enable_service horizon key mysql swift3 s-proxy s-object s-container s-account
then, using the following variables:
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_DEFAULT_DOMAIN=default
export OS_REGION_NAME='RegionOne'
export OS_PROJECT_NAME='swiftprojecttest1'
export OS_USERNAME='swiftusertest1'
export OS_PASSWORD='testing1'
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_AUTH_STRATEGY='keystone'
export OS_AUTH_URL='http://x.x.x.x:35357/v3'
created containers with content in this particular user/project:
stack@s30:/opt$ swift list c0 -l --lh
   0 2016-10-24 20:43:41 application/octet-stream c0_1/
 396 2016-10-24 20:44:05 application/octet-stream c0_1/chcas
then, created EC2 credentials for this user:
stack@s30:/opt$ openstack credential create --type ec2 --project swiftprojecttest1 swiftusertest1 '{"access": "stest1", "secret": "adm1n0"}'
+------------+------------------------------------------------------------------+
| Field      | Value                                                            |
+------------+------------------------------------------------------------------+
| blob       | {"access": "stest1", "secret": "adm1n0"}                         |
| id         | 02bbe6c2553023295717afab00b0a29b9149accbc1f62efb5972e61ced635a07 |
| project_id | c4ac2f50e28644fc8f2f4fabceec2406                                 |
| type       | ec2                                                              |
| user_id    | 4938eb5a49a043eb817e81a43f255cff                                 |
+------------+------------------------------------------------------------------+
stack@s30:/opt$ openstack project list
+----------------------------------+--------------------+
| ID                               | Name               |
+----------------------------------+--------------------+
| c4ac2f50e28644fc8f2f4fabceec2406 | swiftprojecttest1  |
+----------------------------------+--------------------+
stack@s30:/opt$ openstack user list
+----------------------------------+----------------+
| ID                               | Name           |
+----------------------------------+----------------+
| 4938eb5a49a043eb817e81a43f255cff | swiftusertest1 |
+----------------------------------+----------------+

and proxy-server.conf contains configuration for S3 support:

[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit crossdomain swift3 s3token  authtoken keystoneauth tempauth  formpost staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging  proxy-server

[filter:s3token]
paste.filter_factory = keystonemiddleware.s3_token:filter_factory
auth_uri = http://x.x.x.x/identity_admin
cafile = /opt/stack/data/ca-bundle.pem
admin_user = swift
admin_tenant_name = service
admin_password = admin_password

[filter:swift3]
use = egg:swift3#swift3
location = RegionOne

BUT when trying to access this container using s3curl, I'm getting empty list (note changed brackets in output):

doka.ua$ ./s3curl.pl --id stest http://s30.yyyy.zzz:8080
[?xml version='1.0' encoding='UTF-8'?]
[ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"][Owner]
[ID]swiftprojecttest1:swiftusertest1[/ID]
[DisplayName]swiftprojecttest1:swiftusertest1[/DisplayName]
[/Owner][Buckets/][/ListAllMyBucketsResult]

WHERE s3curl configured in this way:

my @endpoints = ( 's30.yyyy.zzz' );
my %awsSecretAccessKeys = (
    stest => {
        id => 'stest1',
        key => 'adm1n0',
    },
);

As you see, swift-proxy successfully authenticates the user but returns empty list while "swift list" returns some content in the container.

Logs between two requests (s3 and swift) show the notable difference when proxy-servers sends final request for data (after all auth sections are successfull):

data request for S3 request looks completely malformed:

proxy-server: 213.160.155.212 213.160.155.212 25/Oct/2016/07/47/00 GET / HTTP/1.0 200 - curl/7.43.0 - - 261 - txc4c3557ea3b84ebd958c7-00580f0deb - 8.8833 - - 1477381611.728308916 1477381620.611623049 -
while this one is for direct "swift list" (with same credentials, x.x.x.x is address of localhost where "swift" was called) returns result as expected:

proxy-server: x.x.x.x x.x.x.x 25/Oct/2016/07/49/43 GET /v1/AUTH_c4ac2f50e28644fc8f2f4fabceec2406%3Fformat%3Djson%26marker%3Dc0 HTTP/1.0 200 - python-swiftclient-3.1.0 ac9b85c65ea1... - 2 - txe6e2c25b2ae64aff99ab2-00580f0e97 - 0.0158 - - 1477381783.317183971 1477381783.332990885 -

Please, suggest where to dig in order to solve this problem.

Thank you!

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2016-11-15 07:46:38 -0500

doka.ua gravatar image

To set logging of full URLS, use "force_swift_request_proxy_log = true" in proxy-server.conf (https://github.com/openstack/swift3/blob/1.11/etc/proxy-server.conf-sample#L110-L118 (https://github.com/openstack/swift3/b...))

In ny case, problem was with the incorrect naming of containers - I used too short names (e.g. c0). By default, Swift use the naming requirements for non-US-East regions, so the name "c0" seems to be invalid; it looks like the bucket name is too short. From S3's docs (http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules (http://docs.aws.amazon.com/AmazonS3/l...)) -

The rules for DNS-compliant bucket names are:

  • Bucket names must be at least 3 and no more than 63 characters long.
  • ...

Try either set the dns_compliant_bucket_names option to False or use longer (>3 characters) for container names.

Thanks to SWIFT Team for help on this issue.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-10-24 16:28:25 -0500

Seen: 526 times

Last updated: Nov 15 '16