arp requests from compute nodes bogging network
My openstack installation is semi-down at the moment due to high network traffic. Upon doing a TCPDUMP from the head-node, I get the following:
15:47:01.732351 IP node-32 > node-20: GREv0, key=0x4, length 50: ARP, Request who-has 192.168.1.2 tell 192.168.1.1, length 28
15:47:01.732378 IP node-32 > node-20: GREv0, key=0x4, length 50: ARP, Request who-has 192.168.1.2 tell 192.168.1.1, length 28
15:47:01.732419 IP node-19 > node-20: GREv0, key=0x4, length 50: ARP, Request who-has 192.168.1.2 tell 192.168.1.1, length 28
15:47:01.732422 IP node-19 > node-20: GREv0, key=0x4, length 50: ARP, Request who-has 192.168.1.2 tell 192.168.1.1, length 28
15:47:01.732506 IP node-19 > node-20: GREv0, key=0x4, length 50: ARP, Request who-has 192.168.1.2 tell 192.168.1.1, length 28
15:47:01.732509 IP node-19 > node-20: GREv0, key=0x4, length 50: ARP, Request who-has 192.168.1.2 tell 192.168.1.1, length 28
The network is saturated (about 1,800,000 packets in 5 seconds) with this kind of traffic. Can someone please help me diagnose what is going on?
I'm using neutron with GRE tunneling. I'm also using a ceph backend.
Thanks