Is Swift appropriate for storing 500 million files each one between 20K to 50K for an application with very high write and read rates? Assuming we use multiple containers with each one holding a subset of the files, where are the performance bottlenecks for this use case?
Mercado Libre shared their case study at the OpenStack summit this week, video available @ http://youtu.be/EKsO2eiAdA8.
As mentioned in their presentation, they have an average image size of 20K, and >1B of them today working very well in their high-write scenarios.
The filesystems on your data disks will start being the limiting factor pretty quickly. The best way to mitigate that is to try to keep inodes and directories in memory as much as possible, which you can do by setting the kernel's vfs_cache_pressure sysctl to 1, and also having plenty of RAM in the object-server boxes.
Asked: 2013-11-06 10:59:24 -0500
Seen: 487 times
Last updated: Nov 11 '13
