Replacing scp with bbcp to obtain fast vm migration (10Gb/sec)

asked 2019-10-09 06:08:01 -0600

elliotp gravatar image

updated 2019-10-09 06:08:40 -0600

Problem: We use local storage for instances. Sometimes an instance can reach 1TB root disk. When migrating such instances between compute nodes it can take many hours.

We use 2 x 10Gb/sec bonded network, enterprise SSDs in RAID 10.

Solution: After testing several solutions (including manual netcat, HPN-SSH), we have found 'bbcp' ( - using bbcp with the default of 4 streams, we can obtain 1.1GB/sec of data transfer (saturating a 10Gb/sec link and most likely the SSD write speed on the target host). We have successfully migrated a 1TB file in 16 minutes.

bbcp could be used as a drop in replacement for scp for migrations, as it runs over SSH, and 'just works'.

Question: How would one have OpenStack's migration feature use bbcp rather than scp? Solutions such as:

  • A configuration parameter to provide the migration vehicle of bbcp rather than scp.
  • Editing the source code somewhere to call bbcp rather than scp.
  • Possibly aliasing scp on the command line to call bbcp instead (although i wanted to avoid this and use it for migrations only).

If anyone has any insight on the above it would be greatly appreciated.

edit retag flag offensive close merge delete