Ask Your Question

Upload images in small chunks

asked 2018-06-05 10:43:31 -0500

mike11 gravatar image

Hi Everyone,

Is there a way to upload large images in small chunks? My issue is that i have only 1GB in cashe memory and when i upload large images (around 3.5 GB) the Linux starts to use swap which slowes the system dramatically. Especially since it is swapping OpenStack services and not only the image.

Thank you for any input, Michael

edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted

answered 2018-06-05 17:15:02 -0500

updated 2018-06-05 17:17:48 -0500

There is only one API for uploading image data (description), and it doesn’t seem to allow uploading in pieces:

curl -i -X PUT -H "X-Auth-Token: $token" -H "Content-Type: application/octet-stream" \ 
     -d @yoursourcefile.qcow2 $image_url/v2/images/{image_id}/file

You don’t describe your setup. How is your cloud configured; is it an all-in-one server? What storage backend do you use for Glance? From where do you upload the image? If you currently do everything on the same server that just has 1GB of cache, perhaps you gain something by keeping the source file and running the openstack CLI command on a separate PC.

Or you could stream the source file from somewhere else:

<program generating image data slowly> | curl -X PUT -H ... -d @- $image-url...

(that’s dash d at dash; I haven’t tried this)

Or you could ease the stress on the cache by throttling the network interface you use for uploading.

Of course, 1GB is woefully small, but it seems to me that there is a lot of experimentation you could do.

edit flag offensive delete link more


Thank you for your reply :) my setting is an all-in-one (which is why i am so limited in space), backend = sqlalchemy. i haven't yet tried your solution, but is this a solution using CLI. is there a way to change to config so i can use the Horizon? Thank you

mike11 gravatar imagemike11 ( 2018-06-06 08:21:44 -0500 )edit

It's not a CLI solution, it's the raw API.

You can throttle the interface no matter how you access OpenStack. Look for the tc command. Out of the box, Horizon uploads a file located on the PC where the browser runs. I am, however, not aware of a CLI or GUI solution for uploading from stdin.

Bernd Bausch gravatar imageBernd Bausch ( 2018-06-06 09:22:08 -0500 )edit

Correction: At least in Pike, Horizon could upload from a URL. This doesn't help you, though, since you can't use this method to control the upload speed.

Bernd Bausch gravatar imageBernd Bausch ( 2018-06-06 09:24:48 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2018-06-05 10:43:31 -0500

Seen: 406 times

Last updated: Jun 05 '18