Ask Your Question

snewpy's profile - activity

2015-07-31 11:39:23 -0600 received badge  Favorite Question (source)
2015-07-15 12:43:40 -0600 received badge  Good Question (source)
2014-04-30 10:22:08 -0600 received badge  Famous Question (source)
2014-04-29 11:02:56 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

The big problem is that failovers with pgpool and postgres are taking a long time because they are waiting for timeouts instead of quickly getting back connection refused.

2014-04-29 11:02:56 -0600 received badge  Commentator
2014-04-29 10:32:46 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

I tried a new kernel (mainline 3.14.2) with the same results.

2014-04-29 09:54:12 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

This is most definitely the rule they are hitting:

[135:5400] -A neutron-openvswi-ia8260acc-e -m state --state INVALID -j DROP
2014-04-29 09:23:06 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

I'm just using telnet. I tried again with netcat, which did set tos to 0x0, but same result other than that. Still the RST packets are blocked somewhere between qbra8260acc-e8 and tapa8260acc-e8.

2014-04-29 08:47:41 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

I think you have to click the (more) button to see the whole question.

2014-04-29 08:40:55 -0600 received badge  Notable Question (source)
2014-04-29 08:36:35 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

I updated the question with the output of the tcpdump on that interface.

2014-04-29 08:31:21 -0600 commented answer iptables INVALID rule preventing RST packets on closed ports between VMs

I posted a wireshark excerpt above, there doesn't seem to be any obvious reason why it is invalid. They are valid enough to reach the qbra8260acc-e8 interface (associated with vm2), but do not reach the tap interface for vm2.

2014-04-29 07:17:45 -0600 commented answer iptables INVALID rule preventing RST packets on closed ports between VMs

I'm not trying to spoof MAC or IP addresses, and I have network connectivity other than RST packets, so I think this is not the same problem. The interfaces in the instances' XML files don't contain any filterref nodes, so I don't think that changing these will help.

2014-04-29 06:56:59 -0600 received badge  Editor (source)
2014-04-29 06:51:24 -0600 commented answer iptables INVALID rule preventing RST packets on closed ports between VMs

I tried removing no-ip-spoofing and no-arp-spoofing from nwfilter in libvirt but it didn't make any difference. My problem isn't pinging or general network connectivity, it's specifically that valid, related RST packets are being marked as INVALID by netfilter.

2014-04-29 06:41:15 -0600 commented question iptables INVALID rule preventing RST packets on closed ports between VMs

I am using the OVS agent. Looking with Wireshark I can see the packet leaving the interface of vm1 and it looks OK, but it gets dropped as invalid by the compute node. I'm not sure what else I can see with Wireshark? By the time it hits tapa8260acc-e8 associated with vm2 it is marked invalid by nf.

2014-04-29 04:30:16 -0600 received badge  Popular Question (source)
2014-04-28 23:23:24 -0600 received badge  Nice Question (source)
2014-04-28 16:21:53 -0600 received badge  Student (source)
2014-04-28 14:52:53 -0600 asked a question iptables INVALID rule preventing RST packets on closed ports between VMs

I have two VMs on a single tenant network (10.5.0.0/24), lets call them vm1 and vm2. Both have the same security groups.

When something is listening on tcp/5432 on vm1, connections from vm2 to that port are successful as expected. However, if nothing is listening, then the RST packet responding to the closed port is dropped by the INVALID rule of vm2's physdev-in chain, making connections time out rather than being refused.

If I replace the DROP action of the INVALID rule with a LOG action, the RST packet gets through and the following packet is logged by netfilter:

IN=qbra8260acc-e8 OUT=qbra8260acc-e8 PHYSIN=qvba8260acc-e8 PHYSOUT=tapa8260acc-e8 MAC=fa:16:3e:12:38:32:fa:16:3e:91:4e:7a:08:00 SRC=10.5.0.42 DST=10.5.0.43 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=22888 DF PROTO=TCP SPT=5432 DPT=55265 WINDOW=0 RES=0x00 ACK RST URGP=0

UPDATE: An additional data point -- these packets are only marked as invalid if the two VMs are running on the same compute node. If they are on different compute nodes, then the RST is not marked invalid.

vm1 (10.5.0.42)'s physdev-out iptables rules are:

-A neutron-openvswi-sg-chain -m physdev --physdev-out tap458d9c1a-32 --physdev-is-bridged -j neutron-openvswi-i458d9c1a-3
-A neutron-openvswi-i458d9c1a-3 -m state --state INVALID -j DROP
-A neutron-openvswi-i458d9c1a-3 -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-i458d9c1a-3 -p icmp -j RETURN
-A neutron-openvswi-i458d9c1a-3 -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -p udp -m udp -m multiport --dports 1:65535 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -s 10.5.0.44/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -s 10.5.0.43/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -s 10.5.0.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-i458d9c1a-3 -j neutron-openvswi-sg-fallback

vm2 (10.5.0.43)'s physdev-in rules are:

-A neutron-openvswi-sg-chain -m physdev --physdev-out tapa8260acc-e8 --physdev-is-bridged -j neutron-openvswi-ia8260acc-e
-A neutron-openvswi-ia8260acc-e -m state --state INVALID -j DROP
-A neutron-openvswi-ia8260acc-e -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-ia8260acc-e -p icmp -j RETURN
-A neutron-openvswi-ia8260acc-e -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-ia8260acc-e -p udp -m udp -m multiport --dports 1:65535 -j RETURN
-A neutron-openvswi-ia8260acc-e -s 10.5.0.44/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-ia8260acc-e -s 10.5.0.42/32 -p tcp -m tcp --dport 5432 -j RETURN
-A neutron-openvswi-ia8260acc-e -s 10.5.0.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-ia8260acc-e -j neutron-openvswi-sg-fallback

This is fresh install of Icehouse on Ubuntu 14.04. Any advice would be very much appreciated.

Wireshark (tshark -i tapa8260acc-e8 'host 10.5.0.42') on the tap interface for vm2 shows no RST packets reaching it, but they do reach the qbra8260acc-e8 interface.

Capturing on 'tapa8260acc-e8'
  1   0.000000    10.5.0.43 -> 10.5.0.42    TCP 74 54667 > postgresql ...
(more)