Ask Your Question
0

VlanManager networking mode, first private bridge interface, and iptables NAT rules

asked 2013-01-16 23:02:07 -0500

xsited gravatar image

I have a Folsom All in One Node and a Compute Node installation using VlanManager networking mode. I need some assistance in finding the source of a (S)NAT rule that is being added that seems erroneous for the my current network topology and causes outbound forwarding issues.

How can iptables corrections be made so they don't have to be made manually after each VM is started?


A brief background on how I arrived here and some more details are provided below.

When I first used nova-manage network to create a range of networks things did not quite work out as planned right away and when I attempted to launch my first VM I ran into the following issue.

https://bugs.launchpad.net/nova/+bug/1076309 (https://bugs.launchpad.net/nova/+bug/...)

I am sure alot of these troubles might have been created by what I did along the way and could have been avoided. But once I deleted all the networks and created new networks VM started correctly again. I wrote a couple of handy script I have included below to create and delete the networks. When the NIC first came up it was assigned a 10.0.0.0/8 adderss, which appears to be the default tenant address and iptables rules accordingly. A restart of the nova processes failed to make the correction, but a reboot did get the private network bridge configured with the correct IP address and subnet mask.

These are rules I need to do execute manually to get the outbound SNAT routes to work properly.

iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -m conntrack ! --ctstate DNAT -j ACCEPT iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.101.0.0/16 -m conntrack ! --ctstate DNAT -j ACCEPT iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.1.1.216/32 -j ACCEPT iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.1.1.216/32 -j ACCEPT iptables -t nat -D nova-network-snat -s 10.0.0.0/8 -o br_ex -j SNAT --to-source 10.4.0.216 iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -o br_ex -j SNAT --to-source 10.4.0.216 iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216

I have flipped through a little bit where this magic might be occurring in nova/network/linux_net.py -- how can these correction be made so I don't have to perform the manually?

Here are some other observation and question I collected along the way. I have not found any specific bugs already reported, but i am happy to file them if warranted.

  1. network-manage create --dns1 and --dns2 did not seem to add anything to the table.

  2. if the network table entries where added they did not seem to be used any way by dnsmasq ...

(more)
edit retag flag offensive close merge delete

6 answers

Sort by ยป oldest newest most voted
0

answered 2013-01-21 04:16:06 -0500

xsited gravatar image

Vish, thanks for your attention to this issue.

From my initial include nova.conf network except I had

routing_source_ip=10.4.0.216

for which I thought the SNAT was being configured from. After some more research I found an issue of similar description.

https://lists.launchpad.net/openstack/msg17025.html (https://lists.launchpad.net/openstack...)

I am still not sure on this one. Evaluation order?

For my question on fixed_range your answer was just what I was asking. Based on the way I was chopping up the class A this one single variable needs to cast a net around the addresses I am using which would be

10.101.0.0/12 which would define a 10.96.0.1 to 10.111.255.254 despite the fact that I was chopping my private networks into /16 networks, if that makes any sense. I guess I coulda simplified it all by using a class B addressing scheme for my private network as well.

I will make this change and retest to see if the 'post VM add and requiring the reapplication of the same SNAT phenomena' is corrected by chance and comeback and close this.

edit flag offensive delete link more
0

answered 2013-01-16 23:29:25 -0500

vishvananda gravatar image

On Jan 16, 2013, at 3:05 PM, xsited question219375@answers.launchpad.net wrote:

https://bugs.launchpad.net/nova/+bug/...

I am sure alot of these troubles might have been created by what I did along the way and could have been avoided. But once I deleted all the networks and created new networks VM started correctly again. I wrote a couple of handy script I have included below to create and delete the networks. When the NIC first came up it was assigned a 10.0.0.0/8 adderss, which appears to be the default tenant address and iptables rules accordingly. A restart of the nova processes failed to make the correction, but a reboot did get the private network bridge configured with the correct IP address and subnet mask.

These are rules I need to do execute manually to get the outbound SNAT routes to work properly.

iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -m conntrack ! --ctstate DNAT -j ACCEPT iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.101.0.0/16 -m conntrack ! --ctstate DNAT -j ACCEPT iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.1.1.216/32 -j ACCEPT iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.1.1.216/32 -j ACCEPT iptables -t nat -D nova-network-snat -s 10.0.0.0/8 -o br_ex -j SNAT --to-source 10.4.0.216 iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -o br_ex -j SNAT --to-source 10.4.0.216 iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216

Simply set:

fixed_range=10.101.0.0/16

in your nova.conf and restart nova-network.

The last two rules appear to duplicate each other so I wouldn't be suprised if you don't need the last one. I'm assuming br_ex contains the default route for the host, if not then you might need to set:

public_interface=xxx

where xxx is the interface that the contains the default route (or a bridge that contains that interface).

Vish

edit flag offensive delete link more
0

answered 2013-01-17 18:47:41 -0500

xsited gravatar image

Vish,

Thanks for your reply. Yes, the public_interface is a bridge with an enslaved interface that provides the default route.

I added the fixed_range variable and restested both after a all nova services restart and a full reboot.

All appears well. I have include the iptables as they appear without any manual modifications. As I went to each of the VMs I could ping the VMs default route (10.101.0.4/16) but not any external routes and I was using 8.8.8.8 as a target. It was not until I manually added this command

iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216

could all the VMs go anywhere they wanted to. Yup, I understand that it appears to be similar to rule 2 of the chain nova-network-snat that already exists.

wrt fixed_range, how will the system and the iptables ruleset be affected by firing up the next tenants network, say 10.102.0.0/16. While the solution you suggested largely cleans up the issue I was experiencing with this network, what additional configuration items, if any, will I need to consider as I am adding VLAN/networks for tenants?

Thanks,

-t

iptables -t nat --line-numbers -n -L

Chain PREROUTING (policy ACCEPT) num target prot opt source destination 1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 2 nova-network-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT) num target prot opt source destination

Chain OUTPUT (policy ACCEPT) num target prot opt source destination 1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 2 nova-network-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain POSTROUTING (policy ACCEPT) num target prot opt source destination 1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 2 MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 3 MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 4 MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 5 nova-network-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 6 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 7 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-api-OUTPUT (1 references) num target prot opt source destination

Chain nova-api-POSTROUTING (1 references) num target prot opt source destination

Chain nova-api-PREROUTING (1 references) num target prot opt source destination

Chain nova-api-float-snat (1 references) num target prot opt source destination

Chain nova-api-snat (1 references) num target prot opt source destination 1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-OUTPUT (1 references) num target prot opt source destination ... (more)

edit flag offensive delete link more
0

answered 2013-01-18 21:47:57 -0500

vishvananda gravatar image

your rule addition seems to be exactly the same. Was it a different source ip originally? If so you may need to set: routing_source_ip=10.4.0.216

The only thing to keep in mind with vlan mode is your fixed_range needs to include all of your vlan networks.

Vish

edit flag offensive delete link more
0

answered 2013-01-27 23:19:12 -0500

xsited gravatar image

The error was the operator...

nova.conf had public_interface=br_ex

while

ifconfig br-ex

br-ex Link encap:Ethernet HWaddr 00:1e:67:4f:bb:26 inet addr:10.4.0.216 Bcast:10.4.7.255 Mask:255.255.248.0 inet6 addr: fe80::21e:67ff:fe4f:bb26/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:16070861 errors:0 dropped:30 overruns:0 frame:0 TX packets:6551736 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:35832866517 (35.8 GB) TX bytes:33707775912 (33.7 GB)

explaining why this was not working

iptables -t nat -A nova-network-snat -s 10.96.0.0/12 -o br_ex -j SNAT --to-source 10.4.0.216

and this made things work until a new instance was added or the system restarted.

iptables -t nat -A nova-network-snat -s 10.96.0.0/12 -j SNAT --to-source 10.4.0.216

doh!

edit flag offensive delete link more
0

answered 2013-05-21 20:41:52 -0500

uictamale gravatar image

For anyone else who arrives here from the link on https://bugs.launchpad.net/nova/+bug/1076309 (https://bugs.launchpad.net/nova/+bug/...) , I actually solved my problem by following the advice on this post: https://privatecloudforums.rackspace.com/viewtopic.php?f=4&t=409 (https://privatecloudforums.rackspace....)

...which says to add a new quota type of fixed_ip. Hope this helps.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-01-16 23:02:07 -0500

Seen: 187 times

Last updated: May 21 '13