VXLAN packet are larger than the standard Ethernet packets. MTU size must be increased to accommodate VXLAN packets. Ensure that all physical switch ports participating in VXLAN network configured to MTU size of 1600. tx-udp_tnl-segmentation which is VXLAN offloading enabled by default. You can find some more details from the following link. Also, your physical Ethernet switch needs to enable jumbo frame. http://www.intel.com/content/www/us/en/embedded/products/networking/xl710-ethernet-controller-vxlan-technical-brief.html (http://www.intel.com/content/www/us/e...)

on our environment we set MTU to 9000, either on compute nodes and on controller nodes, and using vxlan we're close to 10Gbits/sec.

We don't have DVR only VRRP and this is our results:

• east/west on the same vxlan MTU-9000 TCP: 9.80 Gbits/sec
• east/west on the same vxlan MTU-9000 UDP: 8.47 Gbits/sec
• east/west on differt vxlan (traffic through VRRP*): MTU-9000 TCP: 7.36 Gbits/sec

• east/west on differt vxlan (traffic through VRRP*): MTU-9000 UDP: 4.12 Gbits/sec

  • on /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini to increase MTU/performance on VRRP we had to define this parameter:

[agent] ... veth_mtu = 8900

If the MTU in his network was not configured to let 1500+ bytes packets flow then the traffic with "large" packets would just not just affect the performance, it would not let that traffic through at all. So, Paul's MTU configuration across his network is probably fine, he's just experiencing what others have reported: VxLAN performance at default MTU size is poor (partially due to VxLAN overhead). His question still stands. Paul are you monitoring cpu utilization on the hypervisor (for irq's)?

You should really look into VXLAN offloading on both the switches and compute node NICs. I have never personally run a similiar scenario in a lab (VXLAN over 10 Gbit), but have heard about this limitation of cca 6-7 Gbit/s when running on plain vanilla hardware with no additional tuning. Try starting up the benchmarks and have a look at CPU load on your compute nodes, along with number of context switches and both hard and soft interrtupts (dubbed "hi" and "si" in 'top'). I bet your CPUs go crazy.

The even worse performance on 1500B frames could also be explained by the amount of work CPUs are put through. You see, 10 Gbit/s / 1500B gives arount 833k frames per second, with each one causing a hard interrupt and a context switch. This could be mitigated by turning on Generic Recieve Offload on the NIC, which kind of aggregates incoming frames into larger groups.

You can also have a look at Recieve Flow Steering flag or SRIOV functionality of your NIC, which can greatly improve the performance but can rob you of some fancy Nova/Neutron features (live migration, for example).

Hello,

What type of nics are you using and what is the CPU load during the transfer?

Unless you have hardware acceleration for vxlan, everything is done in software which can peg CPUs on the compute nodes.

Depending on the nics being used, you can offload some functions to the card, which can help with throughput. I don't have the list of options available at the moment.

Michael