GRE vs VxLAN tunneling Performance
Hi,
I have a 3 node Openstack Kilo testbed installed - 1x Controller/network and 2 Compute nodes. And I have DVR configured. I am using Iperf3 to test under numerous conditions, but starting point is the tunneling performance using GRE and VXLAN for east/west (vm to vm). Obviously in this scenario with DVR the traffic goes straight from compute1 to compute2. I have these connected to an Arista switch with 10Gbit/sec ports and the servers are Dell poweredge with 10Gbit/sec Intel NICs.
I was expecting that I would get better performance with VxLAN over GRE but when using both 1500-MTU and 9000-MTU, I get better performance from GRE and for example, using VxLAN with 1500-MTU, TCP tests are getting less than 2Gbit/sec.
Here are some approximate test results:
MTU 1500 VXLAN
east/west MTU-1500 TCP = 1.87 Gbits/sec
MTU 9000 VXLAN
east/west MTU-9000 TCP = 5.82 Gbits/sec
MTU 1500 GRE
east/west MTU-1500 TCP = 3.79 Gbits/sec
MTU 9000 GRE
east/west MTU-9000 TCP = 7.86 Gbits/sec
Is this likely to be down to my physical network card or could there be some other factor in these results? I'm pretty sure all my VxLAN config is correct. Its the VxLAN TCP 1500-MTU test that is most surprising.
The Intel network cards I am using don't have the option to enable 'ethtool -K ethX tx-udp_tnl-segmentation' not sure if that is a factor in these results?
thanks in advance, Paul
update:
I have tested with both the following 10Gbit network cards:
Intel Ethernet X540 10Gb BT DP + i350 1Gb BT DP Network Daughter Card
Broadcom 57810 DP 10Gb BT Converged Network Adapter
Whatever I try, I get roughly the same results for east/west traffic (GRE better than VxLAN):
VXLAN
east/west MTU-1500 TCP = 1.87 Gbits/sec
east/west MTU-9000 TCP = 5.82 Gbits/sec
GRE
east/west MTU-1500 TCP = 3.79 Gbits/sec
east/west MTU-9000 TCP = 7.86 Gbits/sec
I have tried extending the MTU for VxLAN to 1600 - no difference in result.
I have tried with and without 'ethtool -K INTERFACE_NAME gro off' - no difference in result.
There is not much difference in CPU usage between GRE and VxLAN.
Open vSwitch version is: 2.3.2, using KVM, OS is Ubuntu.
Paul