Slow virtio network performance
Hi,
I have a grizzly install on ubuntu 13.04 with quantum. The instances are on their private subnet.
Between the instances I see the following network performance:
root@testubi:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43398
[ 5] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43399
[ 6] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43400
[ 7] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43401
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 475 MBytes 398 Mbits/sec
[ 5] 0.0-10.0 sec 662 MBytes 555 Mbits/sec
[ 7] 0.0-10.0 sec 550 MBytes 460 Mbits/sec
[ 6] 0.0-10.0 sec 614 MBytes 514 Mbits/sec
[SUM] 0.0-10.0 sec 2.25 GBytes 1.93 Gbits/sec
[ 8] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43402
[ 4] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43403
[ 5] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43404
[ 6] local 50.50.1.4 port 5001 connected with 50.50.1.5 port 43405
[ 8] 0.0-10.0 sec 705 MBytes 591 Mbits/sec
[ 4] 0.0-10.0 sec 598 MBytes 501 Mbits/sec
[ 5] 0.0-10.0 sec 558 MBytes 468 Mbits/sec
[ 6] 0.0-10.0 sec 575 MBytes 481 Mbits/sec
[SUM] 0.0-10.0 sec 2.38 GBytes 2.04 Gbits/sec
ugh. not very fast. while this is going on, I do see an almost 100% cpu util for the vhostd kernel process on the host.
the server is running virtio for networking:
root@testubi:~# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:05.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:06.0 SCSI storage controller: Red Hat, Inc Virtio block device
root@testubi:~#
any ideas why it would be so slow?
Which plugin and VIF driver are you using? Are the instances on different compute nodes? If so and the plugin is OVS, are they connected with gre tunnels or a provider network?
they are on the same compute node. I'm using the LibvirtHybridOVSBridgeDriver and running openvswitch with quantum.
Maybe try the LibvirtOpenVswitchVirtualPortDriver. And if you test between vms on a linux bridge, do you get the same?