Ask Your Question
0

Can Swift serve up a 301/303 redirect?

asked 2013-02-28 20:44:51 -0500

tnewsome gravatar image

Can I upload a redirect to Swift?

The problem I have is that I have a data store of a lot of files, many of which are identical. When a client requests a file, I would like to redirect that client to the canonical URL for that file (also stored in Swift). This allows clients to maximize their cache hit rate, and prevents them from storing multiple copies of the same thing in their cache.

As a quick example, I'd like to do something like: curl –X PUT -i \ -T JingleRocky.jpg \ https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg (https://storage.swiftdrive.com/v1/CF_...) curl –X PUT -i \ -T /dev/null \ -H "X-Object-Meta-Redirect-303: https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg%22 (https://storage.swiftdrive.com/v1/CF_...) \ https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg (https://storage.swiftdrive.com/v1/CF_...)

Then when I GET https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg (https://storage.swiftdrive.com/v1/CF_...) , Swift would return an HTTP 303 return code, with a redirect to https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg (https://storage.swiftdrive.com/v1/CF_...) .

Is this possible? If not, how much work would you estimate this is to implement by an experienced python programmer? Would such a change be considered for inclusion into the main project?

Thank you, Tim

edit retag flag offensive close merge delete

6 answers

Sort by » oldest newest most voted
0

answered 2013-02-28 22:37:59 -0500

tnewsome gravatar image

Thanks for pointing me towards middleware. That looks like the right solution. Disregard my question about getting it included in core swift. With a flexible system like this there's no need!

edit flag offensive delete link more
0

answered 2013-02-28 22:37:18 -0500

tnewsome gravatar image

Thanks Samuel Merritt, that solved my question.

edit flag offensive delete link more
0

answered 2013-02-28 21:59:07 -0500

torgomatic gravatar image

Note that if you wanted to add this to Swift, you'd almost certainly be told to write it as a middleware, so "write some middleware" does not imply "keep it out of core Swift." The above suggestion makes no statements about the suitability of such a middleware for inclusion in core Swift, either positive or negative.

edit flag offensive delete link more
0

answered 2013-02-28 21:56:42 -0500

torgomatic gravatar image

One could write a custom Swift middleware to handle this. The middleware would pass the request on through to the proxy, and then if (a) the request is a GET for an object and (b) the response headers have X-Meta-Object-Redirect-303 in them, then return a redirect response instead.

edit flag offensive delete link more
0

answered 2013-02-28 21:39:18 -0500

tnewsome gravatar image

I don't think that does what I want. I really do want a 3xx redirect. The issue is that I have clients fetching the same file through a lot of different URLs. So on the server I'll have (with simplified paths) .../project1/baseFile .../project2/baseFile .../project3/baseFile .../canonical/baseFile

I'll have clients getting the baseFile for project[123], and they're really all the same file. These clients are using a standard caching HTTP proxy. Without the redirect, the proxy will end up downloading and keeping 3 identical copies of baseFile in its cache. If project[123]/baseFile redirect to canonical, however, then the proxy will cache the fact that those are redirects (assuming 301), and only download and store the single canonical/baseFile.

I have the same problem in reverse. Many clients are adding files to this database. Each client can quickly check whether a canonical version exists. (This works by putting a hash of the file contents into the name of the canonical file.) Only if it does not, does the client need to upload the canonical version. Once that's done it's quick enough to add a 'redirect' in the appropriate project directory.

Currently I have this setup working with Apache on a single server, but I'd like to use Swift for the reliability and scaling features it provides.

I could try and fix all the clients to not need the redirects, but I don't have as much control over the clients as I do over the server. Another approach is to build something that is a front-end for Swift that does the what I want, but it feels like adding the functionality to Swift directly would not be much extra work, and provide a much better long-term solution.

edit flag offensive delete link more
0

answered 2013-02-28 20:53:59 -0500

notmyname gravatar image

It's an interesting thought, but I'd first be a little concerned about public swift clusters being abused by this. However, that's with about 10 seconds of thought and it may not be a major concern.

Instead of doing 3xx redirects, you can use the existing large-object manifest files for this. You can upload, eg, user_guide_latest.pdf, and have that reference a specific file on the back end (eg versions/user_guide_20130301.pdf). Then you can stage new versions and update the link very quickly.

Docs for large object support: http://docs.openstack.org/developer/s...

--John

On Feb 28, 2013, at 2:46 PM, Tim Newsome question223106@answers.launchpad.net wrote:

New question #223106 on OpenStack Object Storage (swift): https://answers.launchpad.net/swift/+...

Can I upload a redirect to Swift?

The problem I have is that I have a data store of a lot of files, many of which are identical. When a client requests a file, I would like to redirect that client to the canonical URL for that file (also stored in Swift). This allows clients to maximize their cache hit rate, and prevents them from storing multiple copies of the same thing in their cache.

As a quick example, I'd like to do something like: curl –X PUT -i \ -T JingleRocky.jpg \ https://storage.swiftdrive.com/v1/CF_... curl –X PUT -i \ -T /dev/null \ -H "X-Object-Meta-Redirect-303: https://storage.swiftdrive.com/v1/CF_... \ https://storage.swiftdrive.com/v1/CF_...

Then when I GET https://storage.swiftdrive.com/v1/CF_... , Swift would return an HTTP 303 return code, with a redirect to https://storage.swiftdrive.com/v1/CF_... .

Is this possible? If not, how much work would you estimate this is to implement by an experienced python programmer? Would such a change be considered for inclusion into the main project?

Thank you, Tim


You received this question notification because you are a member of Swift Core, which is an answer contact for OpenStack Object Storage (swift).

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-02-28 20:44:51 -0500

Seen: 119 times

Last updated: Feb 28 '13