Revision history [back]

click to hide/show revision 1
initial version

Replacing scp with bbcp to obtain fast vm migration (10Gb/sec)

Problem: We use local storage for instances. Sometimes an instance can reach 1TB root disk. When migrating such instances between compute nodes it can take many hours.

We use 2 x 10Gb/sec bonded network, enterprise SSDs in RAID 10.

Solution: After testing several solutions (including manual netcat, HPN-SSH), we have found 'bbcp' (http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm) - using bbcp with the default of 4 streams, we can obtain 1.1GB/sec of data transfer (saturating a 10Gb/sec link and most likely the SSD write speed on the target host). We have successfully migrated a 1TB file in 16 minutes.

bbcp could be used as a drop in replacement for scp for migrations, as it runs over SSH, and 'just works'.

Question: How would one have OpenStack's migration feature use bbcp rather than scp? Solutions such as:

  • A configuration parameter to provide the migration vehicle of bbcp rather than scp?
  • Editing the source code somewhere to call bbcp rather than scp
  • Possibly aliasing scp on the command line to call bbcp instead (although i wanted to avoid this and use it for migrations only).

If anyone has any insight on the above it would be greatly appreciated.

Replacing scp with bbcp to obtain fast vm migration (10Gb/sec)

Problem: We use local storage for instances. Sometimes an instance can reach 1TB root disk. When migrating such instances between compute nodes it can take many hours.

We use 2 x 10Gb/sec bonded network, enterprise SSDs in RAID 10.

Solution: After testing several solutions (including manual netcat, HPN-SSH), we have found 'bbcp' (http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm) - using bbcp with the default of 4 streams, we can obtain 1.1GB/sec of data transfer (saturating a 10Gb/sec link and most likely the SSD write speed on the target host). We have successfully migrated a 1TB file in 16 minutes.

bbcp could be used as a drop in replacement for scp for migrations, as it runs over SSH, and 'just works'.

Question: How would one have OpenStack's migration feature use bbcp rather than scp? Solutions such as:

  • A configuration parameter to provide the migration vehicle of bbcp rather than scp? scp.
  • Editing the source code somewhere to call bbcp rather than scpscp.
  • Possibly aliasing scp on the command line to call bbcp instead (although i wanted to avoid this and use it for migrations only).

If anyone has any insight on the above it would be greatly appreciated.