Root Cause

This issue is caused by the iptables setup in the reference OVS implementation in Neutron.

Each VM gets its own filtering bridge, so the path of a packet between two VMs on the same host looks like this:

VM1 -> bridge1 (iptables filtering) -> OVS -> bridge2 (iptables filtering) -> VM2

In this setup each packet goes through a conntrack lookup twice (once on each bridge). This would normally not be an issue; however, the conntrack state is shared between the filtering bridges. This is normally not a problem because conntrack is keeping track of both sides of the TCP connection. The issue comes with the RST flag.

When conntrack encounters a TCP packet with a RST flag it immediately destroys the conntrack entry for that connection. This means that once the RST packet reaches the second filtering bridge, the conntrack state has already been removed, so the RST packet is marked as INVALID.

VM1 -> bridge1 (iptables filtering) -> OVS -> bridge2 (iptables filtering) -> VM2
RST >> conntrack destroys conn.     >>>>>>>>> no match, INVALID DROP

If you run conntrack -E -o timestamp while attempting to make a connection that causes a RST, you can see the RST is destroying the state in conntrack:

~$ sudo conntrack -E -o timestamp
[1438290214.284944]     [NEW] tcp      6 120 SYN_SENT src=10.0.0.9 dst=10.0.0.10 sport=36397 dport=99 [UNREPLIED] src=10.0.0.10 dst=10.0.0.9 sport=99 dport=36397 zone=1
[1438290214.285129] [DESTROY] tcp      6 src=10.0.0.9 dst=10.0.0.10 sport=36397 dport=99 [UNREPLIED] src=10.0.0.10 dst=10.0.0.9 sport=99 dport=36397 zone=1

The Fix

There is a bug open for this behavior here: https://bugs.launchpad.net/neutron/+bug/1478925 (https://bugs.launchpad.net/neutron/+b...)

However, it won't be fixed for Icehouse because it's already EOL. It will be fixed in Liberty, but the ability to be back-ported to Juno and Kilo will depend on how complex the fix is.

This can be fixed with conntrack zones, which were only added Kilo so if that's the route taken it won't make it to Juno. It can also be fixed with a hack to get iptables to skip the DESTROY phase, but that will then leave TCP states open that were RST until they expire, so it's not likely that will be an acceptable solution.

I have detailed answer to disable IPTables rules, please refer to link

Disable neutron security group rules then, modify nwfilter nova-base. I deleted no-ip-spoofing and no-arp-spoofing.

@HOST

$ virsh nwfilter-dumpxml nova-base <filter name="nova-base" chain="root"> <uuid>83f28c40-429a-4e50-80a7-0a2b70a2d210</uuid> <filterref filter="no-mac-spoofing"/> <filterref filter="no-ip-spoofing"/> <filterref filter="no-arp-spoofing"/> <filterref filter="allow-dhcp-server"/> </filter>

$ virsh nwfilter-edit nova-base Network filter nova-base XML configuration edited.

$ virsh nwfilter-dumpxml nova-base <filter name="nova-base" chain="root"> <uuid>83f28c40-429a-4e50-80a7-0a2b70a2d210</uuid> <filterref filter="no-mac-spoofing"/> <filterref filter="allow-dhcp-server"/> </filter>

I did ping to vm1 on vm2 again. This time, it succeeded.

Thanks.

Hi, I'm having the same problem. It all comes from the conntrack

RST Packet is marked as INVALID because there is no related line for it in the conntrack.

When placing the VM's on different compute nodes, the SYN packet adds a new record in the conntrack with SYN_SENT value and UNREPLIED state. Once the RST packet returns it related to this flow and everything is O.K.

When placing the VM's on the same compute node. the SYN packet is not tracked by the conntrack and when the RST packet returns it is not related to any existing flow and hence dropped!!!

I used - watch -n 0.5 'cat /proc/net/nf_conntrack | grep My_port_XX'

I'm running with kernel version 2.6.32

I don't see this with ubuntu compute node and cirros instances and linuxbridge-agent. Instead I see the RST packet hit the RELATED,ESTABLISHED rule. You could use Wireshark to look closer at the packets, and maybe see why the return hits the INVALID rule.

$ tcpdump -tvvv -r dump-reset 
reading from file dump-reset, link-type EN10MB (Ethernet)
IP (tos 0x0, ttl 64, id 56219, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.5.57436 > 10.0.0.4.9000: Flags [S], cksum 0x1437 (incorrect -> 0x16a1), seq 4247068559, win 14600, options [mss 1460,sackOK,TS val 377876 ecr 0,nop,wscale 2], length 0
IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    10.0.0.4.9000 > 10.0.0.5.57436: Flags [R.], cksum 0x7b8d (correct), seq 0, ack 4247068560, win 0, length 0