Ask Your Question

Slow iperf performance between vms

asked 2015-06-10 08:16:29 -0600

emilec gravatar image


I am seeing a marked performance difference using iperf between host-host and vm-vm and was hoping someone could help me understand why this is.

I have OpenStack Juno with Neutron (vxlan) running on 2 CentOS 7 hosts. I followed the (OpenStack documentation). One host is responsible for all the services (MariaDB, keystone etc.) as well as compute, the second host is compute only. I have an Intel X710 in each host connected back-to-back. I am using this connection as the instance network. The MTU on the physical NIC is 9000. On a host-host test iperf returns: 16.7 Gbits/sec. On an MTU of 1500 I get 8.71 Gbits/sec.

When running the same iperf test vm-vm (each on a different host) I consistently get somewhere between 1.4 and 1.6 Gbits/sec. The MTU on the vms is 1454. I would expect at least something close to the 8.71 I get on host-host with that MTU size? I've confirmed with tcpdump that the iperf traffic from the vm is flowing over the X710. I've tried turning gro off on the host X710 interface, but this makes no difference to the host-host iperf or the vm-vm iperf.

I've seen suggestions that this is (to be expected), but was wondering if anyone else had any input?

I'd also like to know if anyone has a reference on how you set the MTU sizes for the instance networking that openvswitch creates or do I just set this manually after openvswitch creates all the interfaces? They are all currently set to 1500.

Thanks in advance. Emile

edit retag flag offensive close merge delete


i wonder if you get the same perf with UDP instead of TCP?

On the MTU thing, this was my problem as well. in the [ovs] section of your ml2 conf, add veth_mtu = 1600

don gravatar imagedon ( 2015-06-13 09:39:19 -0600 )edit

make sure network_device_mtu in nova.conf is >= mtu in instance

darragh-oreilly gravatar imagedarragh-oreilly ( 2015-06-15 11:32:52 -0600 )edit

2 answers

Sort by ยป oldest newest most voted

answered 2015-06-13 09:41:07 -0600

don gravatar image

I wonder if the iperf gives u different results with 'udp' than tcp? This could indicate a re-ordering issue or a path-mtu-discovery issue or something like that.

On the MTU thing, this was my problem as well. in the [ovs] section of your ml2 conf, add veth_mtu = 1600 i did that, bumped the outer mtu as you did, and then didn't bother lowering the inner mtu on the instances. Be aware that some network devices have very little buffering for 9K. I used 2K outer MTU for this reason (e.g. 'jumbo' is different than merely 'large').

edit flag offensive delete link more

answered 2015-06-15 10:33:39 -0600

i have found that setting the MTU to 1454 on the instances helps

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2015-06-10 08:15:12 -0600

Seen: 1,285 times

Last updated: Jun 15 '15