Ask Your Question
0

Problem with folders/directories while using S3 middletier

asked 2011-05-05 11:35:24 -0500

lukaszbachman gravatar image

Hi all, I'd like to use Swift as a S3 compatible interface. To do that, we have been using middletier written in Python, provided with swift. The problem I had encountered is happening, when I'm trying to upload an object with path, let's say: bucketname/a/b/c.txt If directories 'a' and 'b' does not exist, the file will fail to upload with status message: <error> InvalidURI <message>Could not parse the specified URI</message> </error>

On the other hand, if I upload file directly to the bucket: bucketname/c.txt everything works as expected.

I guess this is somehow related to this topic in documentation: http://docs.openstack.org/cactus/openstack-object-storage/developer/content/pseudo-hierarchical-folders-directories.html (http://docs.openstack.org/cactus/open...)

But that bug itself puts compatibility with S3 in doubt, since S3 behaves properly in that case - forces creating directories.

Any help would be appreciated, since this is extremely important case for me.

Thanks.

P.S. I'm using S3curl for tests.

edit retag flag offensive close merge delete

9 answers

Sort by ยป oldest newest most voted
0

answered 2011-05-05 20:32:10 -0500

cthier gravatar image

I'm working on it right now, so hopefully today or tomorrow :)

edit flag offensive delete link more
0

answered 2011-05-05 20:33:06 -0500

lukaszbachman gravatar image

Chuk, YOU ARE THE MAN :-) Thanks!

edit flag offensive delete link more
0

answered 2011-05-05 15:02:56 -0500

cthier gravatar image

Hi Lukasz,

Does S3curl work for you in your swift cluster? Other's are having issues with getting it to work with the middleware, so I would be curious to hear if you had to do anything special to make it work?

As to the files that you are uploading. is it a file named "a/b/c.txt" or are a and b directories locally?

edit flag offensive delete link more
0

answered 2011-05-05 19:25:59 -0500

lukaszbachman gravatar image

Hi, thanks for a quick reply. I really appreciate it.

When it comes down to using s3curl - I think the only thing I configured was a parameter inside 's3curl.pl' script, which was listing 'known hosts'. The variable was called 'endopoints', if I recall. Check out this post: http://open.eucalyptus.com/wiki/s3curl I'll try to contact my sys admin to ask if that was the only thing that he has configured.

I don't actually get the other question. I'm trying to send local file from my desktop to specified directory (path) on remote Swift. For instance:

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/c.txt Response: 200

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/a/b/c.txt Response: error - InvalidURI

As you can see the only thing that changes is the destination on my swift storage. When I upload straing to bucket 'mybucket' everything is cool, since this object exists. But when I upload to folder 'mybucket/a/b' - I get an error. S3 would accept that request, no biggie - it would create folders 'a' and 'b'. But Swift somehow does not cope with it.

The other interesting thing is - my sys admin contacted me and told me that he tried achieving the same WITHOUT S3Middletier and it worked! So I guess it now look like python script handling Middletier does it wrong...

edit flag offensive delete link more
0

answered 2011-05-05 19:57:01 -0500

cthier gravatar image

Ah yes, I just found the known hosts things a couple of hours ago :)

and now I can reproduce the error that you were seeing. Turns out there is a small bug in the swift3 middleware.

I will attach a bug to this question, and get the fix in soon. Thanks for reporting the issue!


Chuck

edit flag offensive delete link more
0

answered 2011-05-05 20:30:48 -0500

lukaszbachman gravatar image

That's truly great news! Could You please let me know when could I expect this bug to be fixed? The whole decision upon whether to use Swift or not depends on it :-)

edit flag offensive delete link more
0

answered 2011-05-05 20:33:57 -0500

lukaszbachman gravatar image

Edit: Chuck, ofcourse :)

edit flag offensive delete link more
0

answered 2011-05-05 21:13:02 -0500

cthier gravatar image

The fix was just committed to trunk. Let me know if you run into any more issues.

edit flag offensive delete link more
0

answered 2011-05-05 21:14:26 -0500

lukaszbachman gravatar image

Ok, will do. I'll check it out first thing in the morning (in about 9h).

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-05-05 11:35:24 -0500

Seen: 50 times

Last updated: May 05 '11