Troubles with NAT (MASQUERADE): ICMP works, but not UDP/TCP

asked 2014-05-25 12:22:47 -0600

don
327 ●26 ●33 ●44

I have a host with a /29 available (so after removing 2 for broadcast and 1 for IPMI, not a lot of IP address space). So i am looking to MASQUERADE out for my VMs (with no inbound needd). This is an all-in-one setup, running flat network. I added -t nat -A POSTROUTING -o eth0 -j MASQUERADE.

Something interesting happens, my VM's can now ping to the public internet (going through the masquerade), but cannot do TCP or UDP.

The control network of my VM is 172.16.1/24, and it has IP 172.16.1.16. I can ping 172.16.1.1, its default route (and the IP on my br-ex). My eth0 has a public IP (my only way of reaching this host).

what is causing this ability to ping (ICMP echo) but not TCP or UDP? i do not see them @ all on my eth0 interface. E.g. when i tcpdump br-ex, i see the SYN packet I am trying to send from the VM. When i tcpdump on eth0, i do not see the SYN packet (neither w/ internal nor masqueraded IP).

my neutron secgroup is allow all in, allow all out (-1 for protocol).

any suggestion? how would you set up NAT for this setup? I don't really nead the floatingip since there's no way to reach the VM's anyway.

is there a way to disable the snat in case it is interfering?

    ifconfig -a
    br-ex     Link encap:Ethernet  HWaddr e2:27:cd:ae:cf:4c  
              inet addr:172.16.1.1  Bcast:172.16.1.255  Mask:255.255.255.0
              inet6 addr: fe80::86a:33ff:fe90:54c7/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:1294 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1729 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:148260 (148.2 KB)  TX bytes:148149 (148.1 KB)

    br-int    Link encap:Ethernet  HWaddr 16:ce:e6:9f:62:40  
              inet6 addr: fe80::c0d:c6ff:fea2:dc98/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:3138 errors:0 dropped:0 overruns:0 frame:0
              TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:543366 (543.3 KB)  TX bytes:648 (648.0 B)

    eth0      Link encap:Ethernet  HWaddr 60:eb:69:3e:97:04  
              inet addr:MYIP  Bcast:MYBCAST  Mask:255.255.255.248
              inet6 addr: fe80::62eb:69ff:fe3e:9704/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:745210 errors:0 dropped:0 overruns:0 frame:0
              TX packets:481915 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:347403469 (347.4 MB)  TX bytes:101782097 (101.7 MB)
              Memory:df6e0000-df700000 

    int-br-ex Link encap:Ethernet  HWaddr 46:30:6f:0d:11:4d  
              inet6 addr: fe80::4430:6fff:fe0d:114d/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500 ...

(more)

edit retag flag offensive close merge delete

Comments

Per ifconfig -a eth0 doesn't have IP, per ovs-vsctl show eth0 is not OVS port of br-ex Would guess that you need one more ethernet interface eth1 :-
$ ovs-ctl add-port br-ex eth0
$ iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
View http://www.karlrupp.net/en/computer/n...
How to set rules

dbaxps ( 2014-05-25 22:39:52 -0600 )edit

eth0 does have an IP (the public one of the box). as an experiment I put the other public IP on br-ex, but i can put eth0 in the br-ex group. I was assuming it would route across here (and it seems it is since ICMP works just fine).

i just tried, adding eth0 to the br-ex group does not change anything. i can still ping the world. but not connect via tcp or udp.

This is how my previous system (virtualbox) was setup fwiw.

don ( 2014-05-26 01:30:41 -0600 )edit

OK. Then br-ex has IP 172.16.1.1 and eth0 has IP which is what ?

dbaxps ( 2014-05-26 01:56:43 -0600 )edit

eth0 is the publicly addressable IPv4 of the device.

the general neutron setup for the public network is as below. when i create a VM on the 'public' network, it gets an IP in 172.16.1.10-172.16.1.254. It can ping 172.16.1.1, and it can ping any address on the public internet (NAT'ing through eth0). But it cannot e.g. ssh to the world or fetch a file with HTTP.

I don't believe it would be correct to put eth0 in the bridge group with br-ex (since they are on separate networks they should not be bridged). But the L3 network stack of the host routes between them.

so i guess i don't understand the mechanism that ICMP works and TCP does not here.

ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex

neutron net-create --tenant-id admin public --shared --provider:network_type flat --provider:physical_network ...

(more)

don ( 2014-05-26 05:03:33 -0600 )edit

Take a look at this Fragmented floating IP pools and multiple AS hack It's double nat hack.

dbaxps ( 2014-05-26 05:12:17 -0600 )edit

see more comments

2 answers

Sort by » oldest newest most voted

answered 2014-05-26 05:31:27 -0600

don
327 ●26 ●33 ●44

updated 2014-05-27 20:22:51 -0600

aha!

The machine was using ufw. This is by default operating on br-ex as well as eth0 (firewall between the two).

ufw interferes with neutron and cannot be used (by default it applies rules to all interfaces). So disable it and place manual iptables rules in for firewalling the host.

The other answer given here is overly complex and not correct... you do not need two interfaces. I have one (eth0) which is the public IP of the 'all-in-one'. I created a single public network in neutron, did NOT bridge this to the eth0, allowing routing to take affect, and placed a NAT rule in place for packets exiting eth0. this works just fine.

edit flag offensive delete link

Comments

I hardly remember what ufw is, but ipv4 iptables firewall should be running . I believe you just update system iptables ( analog /etc/sysconfig/iptables) and do service iptables restart

dbaxps ( 2014-05-26 05:55:24 -0600 )edit

OK. Per https://wiki.ubuntu.com/Uncomplicated...
The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls
Then it's supposed to run.

dbaxps ( 2014-05-26 07:51:25 -0600 )edit

On Fedora 20 I have situation like " filerewalld vs ipv4 iptables firewall", so I do
$ service iptables save
$ systemctl disable firewalld
$systemctl enable iptables
$ systemctl stop firewalld
$ systemctl start iptables
Then I have to tune /etc/sysconfig/iptables to open ports for openstack services and for neutron in particular:-

root@icehouse1 ~(keystone_admin)]# iptables-save | grep neutron_server
    -A INPUT -s 192.168.1.127/32 -p tcp -m multiport --dports 9696 -m comment --comment "001 neutron server incoming neutron_server_192.168.1.127_192.168.1.127" -j ACCEPT
    -A INPUT -s 192.168.1.137/32 -p tcp -m multiport --dports 9696 -m comment --comment "001 neutron server incoming neutron_server_192.168.1.127_192.168.1.137" -j ACCEPT

dbaxps ( 2014-05-27 21:30:01 -0600 )edit

Actually, this changes have been done for me by packstack --answer-file=multi-node.txt run. I just had ipv4 iptables firewall up and running before doing packstack setup.

dbaxps ( 2014-05-27 21:39:57 -0600 )edit

add a comment

answered 2014-05-26 23:29:19 -0600

dbaxps

10748 ●57 ●111 ●223 http://bderzhavets.blo...

I believe box has to have eth0 and eth1

1.Interface 'eth1' should be connected to the local net
Interface 'eth0' should be connected to the public network ( not Neutron L3  external subnet)
AIO install should be bind to IP of eth1 subnet ( 172.16.1.0/24)

After setup :-
$ ovs-ctl add-port br-ex eth1
So, eth1 become OVS port of br-ex with no IP.

If just one command
$ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
won't help, then try "double nat hack" :-

/etc/neutron/l3_agent.ini should have
gateway_external_network_id = neutron-public-network-id

start on neutron-l3-agent

script
 iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
 ip addr add 172.16.1.0/24 dev br-ex
 while read private public ; do
   test "$public" || continue
     iptables -t nat -A POSTROUTING -s $private/32 -j SNAT --to-source $public
     iptables -t nat -A PREROUTING  -d $public/32 -j DNAT --to-destination $private
 done <<EOF
 your-floating-ip  public-ip-you-need 
 EOF
 your-floating-ip belongs 172.16.1.0/24

2.If it won't work then manage like suggested in

http://www.andrewklau.com/getting-sta...

Just update for one box ( Andrew did for Cluster ) . View Andrew's answer file solution :-

https://gist.githubusercontent.com/an...

CONFIG_NOVA_API_HOST=172.16.1.11
CONFIG_NOVA_CERT_HOST=172.16.1.11
CONFIG_NOVA_VNCPROXY_HOST=192.168.0.11 (IP from public network)
CONFIG_NOVA_COMPUTE_HOSTS=172.16.1.11
CONFIG_NOVA_CONDUCTOR_HOST=172.16.1.11

I realise that second approach is based on RH's puppet technology and requires translation to Ubuntu.

edit flag offensive delete link

add a comment

Troubles with NAT (MASQUERADE): ICMP works, but not UDP/TCP

Comments

2 answers

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Troubles with NAT (MASQUERADE): ICMP works, but not UDP/TCP edit

Comments

2 answers

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Troubles with NAT (MASQUERADE): ICMP works, but not UDP/TCP