Swift Bulk Operations Best Practices

asked 2017-08-01 17:32:02 -0600

RogerG gravatar image

I have tested uploading objects into swift with a multi-threaded app that made individual requests to create objects. With 20 threads, I was averaging ~6 objects per second per thread. The math showed that it was going to take quite a long time to finish. I turned to bulk operations, and now have a multi-threaded app running that uploads tar.gz files which include the files in their respective containers. It works, but it is slower than what I had running with individual object requests. I am running 10 threads that are each uploading a tar.gz with 4000 objects. Those 10 threads are running at a rate closer to ~2 objects per second per thread. Seems there is something I must be doing wrong.

It seems that swift receives the files within 5 to 10 seconds, but spends 300 to 1600 seconds uncompressing and placing the objects in their containers. I am not positive on that, it is based on watching the network traffic on the machine that is uploading to swift.

Thinking of factors that might impact performance:

  • objects created in a single container versus each object going to a different container
  • number of objects per bulk operation
  • number of concurrent bulk operations
  • bulk file type tar, gz
  • compression level ( full or none )

What are the best practices for this kind of operation?

edit retag flag offensive close merge delete