OVS has high cpu usage, experiencing packet drops

asked 2016-12-27 16:03:36 -0600

samb gravatar image

updated 2016-12-28 09:40:49 -0600

After a power outage, we started experience problems with OVS and networking on our compute nodes. The compute nodes are running Centos 7 and the Mitaka release of openstack with Neutron.

What appears to be happening is that OVS keeps re-adding the ethernet interface bridge. Because of the network problems, I cannot post the exact output logs, so forgive me for brevity.

Our bridge is "em2" and the "ovs-vswitchd.log" file show shows hundreds of lines like:

bridge em2: added interface em2 on port 65534

All the timestamps are milliseconds away. OVS is just constantly doing this. The contents of "journalctl -xe" support this because they show

device em2 has entered promiscuous mode
device em2 has left promiscuous mode

again, just over-and-over, hundreds of lines.

Finally, if I spam the command "ip link show em2" I will see it alternating between:

em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000

and

em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000

The packet drops can be seen as closely as at the physical switch to which the compute nodes are connected. I do not believe there is a problem with the switch itself, because another machine of our is connected to it and it is not experiencing any problems with OVS or packet loss.

ISSUE HAS BEEN FIXED

I guess one of our ifcfg-* files was incorrect. In "ovs-vsctl show" we had "em2" as its own bridge with no ports except itself. All we had to do was "ovs-vsctl del-br em2" and the issue was resolved.

So I think what was happening was that the interface was being added to the em2 bridge, which was not connected to anything so packets would be lost, then it would be disconnected to the bridge, then a packet would go through, then it would attempt to reconnect it to the bridge, and so on. I really don't understand OVS so I can't provide an accurate description, but hopefully this will help someone.

edit retag flag offensive close merge delete

Comments

I guess the problem is defining interface on OVS local port. The local port is connected to the ipstack of the machine on which OVS is running. So all the packets flows on the wrong route.

fifi gravatar imagefifi ( 2016-12-28 20:18:46 -0600 )edit