Ask Your Question
3

Quantum (neutron) gre slow performance [closed]

asked 2013-10-20 05:15:12 -0500

laboshinl gravatar image

updated 2013-10-20 05:28:49 -0500

I got two nodes Grizzly installation on CentOS 6.4: tn0(192.168.250.20) All openstack services tn1(192.168.250.21) quantum-openvswitch-agent, compute

My problem is that Iperf between tn1 VM's and tn0 VM's or any machines at my subnet (192.168.250.0/23) shows 140 Kb/s

Iperf between tn0 VM's and 192.168.250.0/23 shows 900 Mb/s

Can someone please explain why performance is so slow? Is it mtu problem or something? There are my configs:

[root@tn0 ~]# quantum agent-list
+--------------------------------------+--------------------+-----------------+-------+----------------+
| id                                   | agent_type         | host            | alive | admin_state_up |
+--------------------------------------+--------------------+-----------------+-------+----------------+
| 55225703-f6d8-4a15-8632-83c0dd45ff2c | DHCP agent         | tn1.stu.neva.ru | xxx   | True           |
| 9ffbe892-42cd-409a-a6b2-199946c4def0 | DHCP agent         | tn0.stu.neva.ru | :-)   | True           |
| b261ed57-441e-4489-9fb7-a66c7e410126 | Open vSwitch agent | tn0.stu.neva.ru | :-)   | True           |
| c0d20a83-a5b9-4b39-b063-752a664c3511 | Open vSwitch agent | tn1.stu.neva.ru | :-)   | True           |
| d5dd7b83-10d1-422a-a30a-231368d26e96 | L3 agent           | tn0.stu.neva.ru | :-)   | True           |
+--------------------------------------+--------------------+-----------------+-------+----------------+

[root@tn0 ~]# ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:1E:67:04:28:0B  
          inet addr:192.168.250.20  Bcast:192.168.251.255  Mask:255.255.254.0
          inet6 addr: fe80::5c64:77ff:fe27:3260/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:1383023 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1177310 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:23644682530 (22.0 GiB)  TX bytes:313492896 (298.9 MiB)

br-int    Link encap:Ethernet  HWaddr C2:28:DC:EE:AB:47  
          inet6 addr: fe80::641b:4dff:fe87:aec3/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:18442 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:804564 (785.7 KiB)  TX bytes:468 (468.0 b)

br-tun    Link encap:Ethernet  HWaddr F6:5D:52:AA:3D:41  
          inet6 addr: fe80::a458:cfff:fec3:1dee/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8180 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:353164 (344.8 KiB)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:1E:67:04:28:0B  
          inet6 addr: fe80::21e:67ff:fe04:280b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1546  Metric:1
          RX packets:1055723 errors:49 dropped:0 overruns:0 frame:49
          TX packets:880976 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:329252619 (313.9 MiB)  TX bytes:294162404 (280.5 MiB)
          Interrupt:16 Memory:b1a00000-b1a20000 

eth1      Link encap:Ethernet  HWaddr 00:1E:67:04:28:0A  
          inet addr:192.168.250.22  Bcast:192.168.251.255  Mask:255.255.254.0
          inet6 addr: fe80::21e:67ff:fe04:280a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1546  Metric:1
          RX packets:12007 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes ...
(more)
edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by laboshinl
close date 2013-11-15 04:14:16.054789

Comments

You might be experiencing the same issue that is discussed in the thread on mailing list: http://lists.openstack.org/pipermail/openstack/2013-October/001778.html It seems like there is no known solution and it is a new problem.

dasp gravatar imagedasp ( 2013-10-23 21:10:02 -0500 )edit
add a comment

2 answers

Sort by ยป oldest newest most voted
8

answered 2013-10-24 16:21:20 -0500

JakeWarner gravatar image

I had this issue too and spent almost two months debugging it. I tried tweaking MTUs, enabling jumbro frames and everything. The issue ended up being with GRO where the server was offloading network processing off to the NIC. I disabled that, restarted the interface and my speeds increased by 8x.

Disable gro for an interface (temporary, if the machine restarts, this will be lost) with:

ethtool -K <interface> gro off

You can disable it permanently (on CentOS/RHEL at least) by adding this to the respective interface's network script:

ETHTOOL_OPTS="gro off"

The script (on RHEL/CentOS) is found at:

/etc/sysconfigs/network-scripts/ifcfg-<interface>
edit flag offensive delete link more

Comments

I had a similar experience and this worked great for me. Anyone have any idea of how to make this permanent on Ubuntu?

ctaylor gravatar imagectaylor ( 2013-11-05 16:49:03 -0500 )edit

I'm also experiencing same issue on Ubuntu.

mtasic85 gravatar imagemtasic85 ( 2014-03-07 09:24:37 -0500 )edit
add a comment
2

answered 2013-10-24 14:50:11 -0500

dasp gravatar image

updated 2014-02-27 16:50:28 -0500

smaffulli gravatar image

I've just solved the problem for myself.

Hint: if you want to check if this answer will solve your issue, execute ifconfig eth0 mtu 1400 on the instance and check the speed.

Edit the file: /etc/quantum/dhcp_agent.ini and set:

dnsmasq_config_file=/etc/neutron/dnsmasq-neutron.conf

Create and edit file /etc/neutron/dnsmasq-neutron.conf and put this inside:

dhcp-option-force=26,1400

Restart DHCP:

service neutron-dhcp-agent restart

And reboot your instance. Verify that your MTU is 1400:

 root@web:~# ifconfig eth0      Link
 encap:Ethernet  HWaddr
 fa:16:3e:f0:6a:9f  
           inet addr:10.1.0.2  Bcast:10.1.0.255  Mask:255.255.255.0
           inet6 addr: fe80::f816:3eff:fef0:6a9f/64
 Scope:Link
           UP BROADCAST RUNNING MULTICAST  **MTU:1400**  Metric:1
           RX packets:934855 errors:0 dropped:0 overruns:0 frame:0
           TX packets:207741 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:1210256390 (1.2 GB)  TX bytes:226172592 (226.1 MB)

Test upload speed again. It should be back to normal.

edit flag offensive delete link more

Comments

In this case it's enough to decrease mtu to 1454 dhcp-option-force=26,1454

laboshinl gravatar imagelaboshinl ( 2013-11-15 03:53:52 -0500 )edit

it really works in my environment.thanks

Haifeng gravatar imageHaifeng ( 2013-12-12 21:25:06 -0500 )edit

Changing MTU size to 1454 solved issue with network for me too.

Drankis gravatar imageDrankis ( 2014-07-17 07:11:28 -0500 )edit
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

3 followers

subscribe to rss feed

Stats

Asked: 2013-10-20 05:15:12 -0500

Seen: 5,460 times

Last updated: Feb 27 '14

Related questions

Cause for strange network config?

Use the same Neutron node for both VLAN and GRE traffic

Plugin ml2 configuration for neutron on icehouse, debian wheezy

RDO vlan(ed) network over internal network adapter doesn't work

issue with neutron networking

Using encreypted GRE tunnels

mask expected for non-Ethernet II frame[openvswitch]

Neutron Error: Unable to create tunnel port. Invalid remote IP: INSTANCE_TUNNELS_INTERFACE_IP_ADDRESS [closed]

VMs on subnet suffer IGMP timeouts.

Router External Gateway DOWN