Ask Your Question
0

(glance S3 client) plan to support multipart upload method

asked 2011-10-21 10:38:11 -0600

ozaint gravatar image

Hi,

Is there any plan for glance S3 client to use multipart upload method for the large virtual image file upload in the future release ?

I haven't tested but it looks like that there are CHUNK upload features in Swift Store and RBD Store but nothing in S3 Store. Hopefully, S3 Store should have it as well.

What size, in MB, should Glance start chunking image files

and do a large object manifest in Swift? By default, this is

the maximum object size in Swift, which is 5GB

swift_store_large_object_size = 5120

When doing a large object manifest, what size, in MB, should

Glance write chunks to Swift? This amount of data is written

to a temporary disk buffer during the process of chunking

the image file, and the default is 200MB

swift_store_large_object_chunk_size = 200

Images will be chunked into objects of this size (in megabytes).

For best performance, this should be a power of two

rbd_store_chunk_size = 8

edit retag flag offensive close merge delete

5 answers

Sort by ยป oldest newest most voted
0

answered 2011-10-24 20:07:40 -0600

jaypipes gravatar image

Hi!

Actually, the only reason we had to do something special for Swift was because it doesn't handle images greater than 5G without manually splitting the file into chunks :)

The current Glance S3 driver actually is extremely inefficient because of the Boto library's design. Because the call to boto.s3.Key.set_contents_from_file() requires a seekable file descriptor (and webob.Request.body_file is not seekable by default and setting the seekable attribute to True pulls the entire body into a StringIO in memory.

To prevent this problem, which resulted in OutOfMemory errors when image file sizes exceeded the available memory (common for Windows images), we write the incoming webob.Request.body_file to a temporary file on the API server's disk in chunks. Currently, that chunk size is not configurable, but it sounds like you'd like to make it configurable?

-jay

edit flag offensive delete link more

Comments

Hello ,

I am trying to use Glance S3 , and have issues where it does not show any PUT request. Any help

amit-g gravatar imageamit-g ( 2017-08-14 14:38:25 -0600 )edit
0

answered 2011-10-25 02:09:22 -0600

ozaint gravatar image

Hi,

Thank you for the response.

Well, my point is if Glance can use S3 MultiPart Upload REST API for the large objects. I believe boto already provides this API and Amazon recommends it for the large object upload. If it can be implemented in S3 client in Glance, It's very useful for the customers and a good thing for AWS and other S3 servers.

http://www.elastician.com/2010/12/s3-multipart-upload-in-boto.html (http://www.elastician.com/2010/12/s3-...) http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-upload.html (http://aws.typepad.com/aws/2010/11/am...) http://www.centernetworks.com/amazon-web-services-launches-multipart-s3-uploads (http://www.centernetworks.com/amazon-...)

edit flag offensive delete link more
0

answered 2011-10-25 16:23:23 -0600

jaypipes gravatar image

Thanks for those links, Mozawa! I'll turn this into a blueprint.

Cheers! -jay

edit flag offensive delete link more
0

answered 2011-10-25 16:24:57 -0600

jaypipes gravatar image
edit flag offensive delete link more
0

answered 2011-10-25 16:25:17 -0600

jaypipes gravatar image

Blueprint created

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-10-21 10:38:11 -0600

Seen: 194 times

Last updated: Oct 25 '11