swift performance for small objects

asked 2013-11-06 10:59:24 -0600

anonymous user


Is Swift appropriate for storing 500 million files each one between 20K to 50K for an application with very high write and read rates? Assuming we use multiple containers with each one holding a subset of the files, where are the performance bottlenecks for this use case?

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted

answered 2013-11-07 20:01:32 -0600

mblandini gravatar image

updated 2013-11-11 04:50:27 -0600

koolhead17 gravatar image

Mercado Libre shared their case study at the OpenStack summit this week, video available @ http://youtu.be/EKsO2eiAdA8.

As mentioned in their presentation, they have an average image size of 20K, and >1B of them today working very well in their high-write scenarios.

edit flag offensive delete link more

answered 2013-11-06 13:41:01 -0600

torgomatic gravatar image

The filesystems on your data disks will start being the limiting factor pretty quickly. The best way to mitigate that is to try to keep inodes and directories in memory as much as possible, which you can do by setting the kernel's vfs_cache_pressure sysctl to 1, and also having plenty of RAM in the object-server boxes.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2013-11-06 10:59:24 -0600

Seen: 510 times

Last updated: Nov 11 '13