Ask Your Question
0

Problem with folders/directories while using S3 middletier

asked 2011-05-05 11:35:24 -0500

lukaszbachman gravatar image

Hi all, I'd like to use Swift as a S3 compatible interface. To do that, we have been using middletier written in Python, provided with swift. The problem I had encountered is happening, when I'm trying to upload an object with path, let's say: bucketname/a/b/c.txt If directories 'a' and 'b' does not exist, the file will fail to upload with status message: <error> InvalidURI <message>Could not parse the specified URI</message> </error>

On the other hand, if I upload file directly to the bucket: bucketname/c.txt everything works as expected.

I guess this is somehow related to this topic in documentation: http://docs.openstack.org/cactus/openstack-object-storage/developer/content/pseudo-hierarchical-folders-directories.html (http://docs.openstack.org/cactus/open...)

But that bug itself puts compatibility with S3 in doubt, since S3 behaves properly in that case - forces creating directories.

Any help would be appreciated, since this is extremely important case for me.

Thanks.

P.S. I'm using S3curl for tests.

edit retag flag offensive close merge delete

9 answers

Sort by ยป oldest newest most voted
0

answered 2011-05-05 21:14:26 -0500

lukaszbachman gravatar image

Ok, will do. I'll check it out first thing in the morning (in about 9h).

edit flag offensive delete link more
0

answered 2011-05-05 21:13:02 -0500

cthier gravatar image

The fix was just committed to trunk. Let me know if you run into any more issues.

edit flag offensive delete link more
0

answered 2011-05-05 20:33:57 -0500

lukaszbachman gravatar image

Edit: Chuck, ofcourse :)

edit flag offensive delete link more
0

answered 2011-05-05 20:33:06 -0500

lukaszbachman gravatar image

Chuk, YOU ARE THE MAN :-) Thanks!

edit flag offensive delete link more
0

answered 2011-05-05 20:32:10 -0500

cthier gravatar image

I'm working on it right now, so hopefully today or tomorrow :)

edit flag offensive delete link more
0

answered 2011-05-05 20:30:48 -0500

lukaszbachman gravatar image

That's truly great news! Could You please let me know when could I expect this bug to be fixed? The whole decision upon whether to use Swift or not depends on it :-)

edit flag offensive delete link more
0

answered 2011-05-05 19:57:01 -0500

cthier gravatar image

Ah yes, I just found the known hosts things a couple of hours ago :)

and now I can reproduce the error that you were seeing. Turns out there is a small bug in the swift3 middleware.

I will attach a bug to this question, and get the fix in soon. Thanks for reporting the issue!


Chuck

edit flag offensive delete link more
0

answered 2011-05-05 19:25:59 -0500

lukaszbachman gravatar image

Hi, thanks for a quick reply. I really appreciate it.

When it comes down to using s3curl - I think the only thing I configured was a parameter inside 's3curl.pl' script, which was listing 'known hosts'. The variable was called 'endopoints', if I recall. Check out this post: http://open.eucalyptus.com/wiki/s3curl I'll try to contact my sys admin to ask if that was the only thing that he has configured.

I don't actually get the other question. I'm trying to send local file from my desktop to specified directory (path) on remote Swift. For instance:

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/c.txt Response: 200

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/a/b/c.txt Response: error - InvalidURI

As you can see the only thing that changes is the destination on my swift storage. When I upload straing to bucket 'mybucket' everything is cool, since this object exists. But when I upload to folder 'mybucket/a/b' - I get an error. S3 would accept that request, no biggie - it would create folders 'a' and 'b'. But Swift somehow does not cope with it.

The other interesting thing is - my sys admin contacted me and told me that he tried achieving the same WITHOUT S3Middletier and it worked! So I guess it now look like python script handling Middletier does it wrong...

edit flag offensive delete link more
0

answered 2011-05-05 15:02:56 -0500

cthier gravatar image

Hi Lukasz,

Does S3curl work for you in your swift cluster? Other's are having issues with getting it to work with the middleware, so I would be curious to hear if you had to do anything special to make it work?

As to the files that you are uploading. is it a file named "a/b/c.txt" or are a and b directories locally?

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-05-05 11:35:24 -0500

Seen: 50 times

Last updated: May 05 '11