Ask Your Question
0

What are the parameters called max_manifest_segments,max_manifest_size,min_segment_size under the Filter SLO in swift proxy?

asked 2014-07-04 08:40:06 -0500

ebyenjoys gravatar image

I didn't find any explanations for max_manifest_segments,max_manifest_size,min_segment_size parameters in proxy server configuration of swift ? Could anyone know about these parameters?

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
1

answered 2014-09-03 22:56:41 -0500

SGPJ gravatar image
enter code Middleware that will provide Static Large Object (SLO) support.

This feature is very similar to Dynamic Large Object (DLO) support in that it allows the user to upload many objects concurrently and afterwards download them as a single object. It is different in that it does not rely on eventually consistent container listings to do so. Instead, a user defined manifest of the object segments is used.


Uploading the Manifest

After the user has uploaded the objects to be concatenated a manifest is uploaded. The request must be a PUT with the query parameter::

?multipart-manifest=put

The body of this request will be an ordered list of files in json data format. The data to be supplied for each segment is::

path: the path to the segment (not including account)
      /container/object_name
etag: the etag given back when the segment was PUT
size_bytes: the size of the segment in bytes

The format of the list will be::

json:
[{"path": "/cont/object",
  "etag": "etagoftheobjectsegment",
  "size_bytes": 1048576}, ...]

The number of object segments is limited to a configurable amount, default 1000. Each segment, except for the final one, must be at least 1 megabyte (configurable). On upload, the middleware will head every segment passed in and verify the size and etag of each. If any of the objects do not match (not found, size/etag mismatch, below minimum size) then the user will receive a 4xx error response. If everything does match, the user will receive a 2xx response and the SLO object is ready for downloading.

Behind the scenes, on success, a json manifest generated from the user input is sent to object servers with an extra "X-Static-Large-Object: True" header and a modified Content-Type. The parameter: swift_bytes=$total_size will be appended to the existing Content-Type, where total_size is the sum of all the included segments' size_bytes. This extra parameter will be hidden from the user.

Manifest files can reference objects in separate containers, which will improve concurrent upload speed. Objects can be referenced by multiple manifests. The segments of a SLO manifest can even be other SLO manifests. Treat them as any other object i.e., use the Etag and Content-Length given on the PUT of the sub-SLO in the manifest to the parent SLO.


Retrieving a Large Object

A GET request to the manifest object will return the concatenation of the objects from the manifest much like DLO. If any of the segments from the manifest are not found or their Etag/Content Length no longer match the connection will drop. In this case a 409 Conflict will be logged in the proxy logs and the user will receive incomplete results.

The headers from this GET or HEAD request will return the metadata attached to the manifest object itself with some exceptions::

Content-Length: the total size of the SLO (the sum of the sizes of
                the segments in the manifest)
X-Static-Large-Object: True
Etag: the etag of the SLO (generated the same way as DLO)

A GET request with the ...

(more)
edit flag offensive delete link more
0

answered 2014-09-03 21:43:34 -0500

zaitcev gravatar image
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-07-04 08:40:06 -0500

Seen: 325 times

Last updated: Sep 03 '14