Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

IPv6 multicast storm

We have an Openstack platform running for 2+ years. The current version is Ocata and it is HA configured with HA proxy.

Two weeks ago there was suddenly a IPv6 multicast storm that took down our network. We did a packet capture and saw a lot of "Router Solicitation" and "Neighbor Advertisement" messages being sent out every 1/1000 ms from some of the routers inside Openstack. The networks connected to the routers are all of IPv4 type. We rebooted the Openstack machines and the problem went away.

We saw some logs at about the time of the incident:

Mar 9 13:38:20 xxxxxxxxxx kernel: ICMPv6: NA: someone advertises our address fe80:0000:0000:0000:f816:xxx:fe85:b2af on qg-xxxxxx-cd!

2018-03-09 13:38:18.424 15230 INFO neutron.agent.l3.ha [-] Router xxxxxxxxxxxx-28a4-47a6-b4f9-3472dc51d4d1 transitioned to backup 2018-03-09 13:38:18.433 15230 INFO eventlet.wsgi.server [-] <local> - - [09/Mar/2018 13:38:18] "GET / HTTP/1.1" 200 115 0.008982 2018-03-09 13:38:19.641 15230 INFO neutron.agent.l3.ha [-] Router xxxxxxxxxxxx-6265-478e-8a69-4631e7acfd93 transitioned to backup 2018-03-09 13:38:19.649 15230 INFO eventlet.wsgi.server [-] <local> - - [09/Mar/2018 13:38:19] "GET / HTTP/1.1" 200 115 0.008043 2018-03-09 13:38:20.905 15230 INFO neutron.agent.l3.ha [-] Router xxxxxxx-4451-43b3-bd6c-9615aee81a2a transitioned to master

We did a patch as per https://review.openstack.org/#/c/460918/2 for backup router having the same MAC addr as the master router. But we are not convinced this was the cause of the multicast storm in the first place.