# Problem with folders/directories while using S3 middletier

Hi all, I'd like to use Swift as a S3 compatible interface. To do that, we have been using middletier written in Python, provided with swift. The problem I had encountered is happening, when I'm trying to upload an object with path, let's say: bucketname/a/b/c.txt If directories 'a' and 'b' does not exist, the file will fail to upload with status message: <error> InvalidURI <message>Could not parse the specified URI</message> </error>

On the other hand, if I upload file directly to the bucket: bucketname/c.txt everything works as expected.

I guess this is somehow related to this topic in documentation: http://docs.openstack.org/cactus/openstack-object-storage/developer/content/pseudo-hierarchical-folders-directories.html (http://docs.openstack.org/cactus/open...)

But that bug itself puts compatibility with S3 in doubt, since S3 behaves properly in that case - forces creating directories.

Any help would be appreciated, since this is extremely important case for me.

Thanks.

P.S. I'm using S3curl for tests.

edit retag close merge delete

Sort by » oldest newest most voted

I'm working on it right now, so hopefully today or tomorrow :)

more

Chuk, YOU ARE THE MAN :-) Thanks!

more

Hi Lukasz,

Does S3curl work for you in your swift cluster? Other's are having issues with getting it to work with the middleware, so I would be curious to hear if you had to do anything special to make it work?

As to the files that you are uploading. is it a file named "a/b/c.txt" or are a and b directories locally?

more

Hi, thanks for a quick reply. I really appreciate it.

When it comes down to using s3curl - I think the only thing I configured was a parameter inside 's3curl.pl' script, which was listing 'known hosts'. The variable was called 'endopoints', if I recall. Check out this post: http://open.eucalyptus.com/wiki/s3curl I'll try to contact my sys admin to ask if that was the only thing that he has configured.

I don't actually get the other question. I'm trying to send local file from my desktop to specified directory (path) on remote Swift. For instance:

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/c.txt Response: 200

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/a/b/c.txt Response: error - InvalidURI

As you can see the only thing that changes is the destination on my swift storage. When I upload straing to bucket 'mybucket' everything is cool, since this object exists. But when I upload to folder 'mybucket/a/b' - I get an error. S3 would accept that request, no biggie - it would create folders 'a' and 'b'. But Swift somehow does not cope with it.

The other interesting thing is - my sys admin contacted me and told me that he tried achieving the same WITHOUT S3Middletier and it worked! So I guess it now look like python script handling Middletier does it wrong...

more

Ah yes, I just found the known hosts things a couple of hours ago :)

and now I can reproduce the error that you were seeing. Turns out there is a small bug in the swift3 middleware.

I will attach a bug to this question, and get the fix in soon. Thanks for reporting the issue!

Chuck

more

That's truly great news! Could You please let me know when could I expect this bug to be fixed? The whole decision upon whether to use Swift or not depends on it :-)

more

Edit: Chuck, ofcourse :)

more

The fix was just committed to trunk. Let me know if you run into any more issues.

more

Ok, will do. I'll check it out first thing in the morning (in about 9h).

more