iptables INVALID rule preventing RST packets on closed ports between VMs
I have two VMs on a single tenant network (10.5.0.0/24), lets call them vm1 and vm2. Both have the same security groups.
When something is listening on tcp/5432 on vm1, connections from vm2 to that port are successful as expected. However, if nothing is listening, then the RST packet responding to the closed port is dropped by the INVALID rule of vm2's physdev-in chain, making connections time out rather than being refused.
If I replace the DROP action of the INVALID rule with a LOG action, the RST packet gets through and the following packet is logged by netfilter:
IN=qbra8260acc-e8 OUT=qbra8260acc-e8 PHYSIN=qvba8260acc-e8 PHYSOUT=tapa8260acc-e8 MAC=fa:16:3e:12:38:32:fa:16:3e:91:4e:7a:08:00 SRC=10.5.0.42 DST=10.5.0.43 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=22888 DF PROTO=TCP SPT=5432 DPT=55265 WINDOW=0 RES=0x00 ACK RST URGP=0
UPDATE: An additional data point -- these packets are only marked as invalid if the two VMs are running on the same compute node. If they are on different compute nodes, then the RST is not marked invalid.
vm1 (10.5.0.42)'s physdev-out iptables rules are:
-A neutron-openvswi-sg-chain -m physdev --physdev-out tap458d9c1a-32 --physdev-is-bridged -j neutron-openvswi-i458d9c1a-3
-A neutron-openvswi-i458d9c1a-3 -m state --state INVALID -j DROP
-A neutron-openvswi-i458d9c1a-3 -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-i458d9c1a-3 -p icmp -j RETURN
-A neutron-openvswi-i458d9c1a-3 -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -p udp -m udp -m multiport --dports 1:65535 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -s 10.5.0.44/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -s 10.5.0.43/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -s 10.5.0.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -j neutron-openvswi-sg-fallback
vm2 (10.5.0.43)'s physdev-in rules are:
-A neutron-openvswi-sg-chain -m physdev --physdev-out tapa8260acc-e8 --physdev-is-bridged -j neutron-openvswi-ia8260acc-e
-A neutron-openvswi-ia8260acc-e -m state --state INVALID -j DROP
-A neutron-openvswi-ia8260acc-e -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-ia8260acc-e -p icmp -j RETURN
-A neutron-openvswi-ia8260acc-e -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-ia8260acc-e -p udp -m udp -m multiport --dports 1:65535 -j RETURN
-A neutron-openvswi-ia8260acc-e -s 10.5.0.44/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-ia8260acc-e -s 10.5.0.42/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-ia8260acc-e -s 10.5.0.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-ia8260acc-e -j neutron-openvswi-sg-fallback
This is fresh install of Icehouse on Ubuntu 14.04. Any advice would be very much appreciated.
Wireshark (tshark -i tapa8260acc-e8 'host 10.5.0.42') on the tap interface for vm2 shows no RST packets reaching it, but they do reach the qbra8260acc-e8 interface.
Capturing on 'tapa8260acc-e8'
1 0.000000 10.5.0.43 -> 10.5.0.42 TCP 74 54667 > postgresql ...
I am using the OVS agent. Looking with Wireshark I can see the packet leaving the interface of vm1 and it looks OK, but it gets dropped as invalid by the compute node. I'm not sure what else I can see with Wireshark? By the time it hits tapa8260acc-e8 associated with vm2 it is marked invalid by nf.
can you show tcpdump -vvv on the qbra8260acc-e8 interface?
I updated the question with the output of the tcpdump on that interface.
I don't see it
I think you have to click the (more) button to see the whole question.