Replacing scp with bbcp to obtain fast vm migration (10Gb/sec)
Problem: We use local storage for instances. Sometimes an instance can reach 1TB root disk. When migrating such instances between compute nodes it can take many hours.
We use 2 x 10Gb/sec bonded network, enterprise SSDs in RAID 10.
Solution: After testing several solutions (including manual netcat, HPN-SSH), we have found 'bbcp' (http://pcbunn.cithep.caltech.edu/bbcp...) - using bbcp with the default of 4 streams, we can obtain 1.1GB/sec of data transfer (saturating a 10Gb/sec link and most likely the SSD write speed on the target host). We have successfully migrated a 1TB file in 16 minutes.
bbcp could be used as a drop in replacement for scp for migrations, as it runs over SSH, and 'just works'.
Question: How would one have OpenStack's migration feature use bbcp rather than scp? Solutions such as:
- A configuration parameter to provide the migration vehicle of bbcp rather than scp.
- Editing the source code somewhere to call bbcp rather than scp.
- Possibly aliasing scp on the command line to call bbcp instead (although i wanted to avoid this and use it for migrations only).
If anyone has any insight on the above it would be greatly appreciated.