Ask Your Question
2

how to configure multiple external networks in RDO Liberty/Mitaka

asked 2016-04-27 15:14:15 -0600

sacha-m gravatar image

I have to install a network device that will do QoS for one of my customers. Due I only have one External network for Openstack (192.168.70.0/24), which leads to (beside others) the Internet, and I don’t want to pass all my traffic (where are other customer’s traffic in there) thru this network device, I decide to create another external network (VLAN), that will be used only for this only customer, and will have the network device (physical) on it. So, it will be something like this: image description So, I have my physical environment configured to pass this “new” traffic thru the VLAN 90, and now I have the linux where I’m installing Openstack RDO Liberty (and also tested on Mitaka) all-in-one with this configuration before configuring any bridge on it:

# cat /etc/sysconfig/network-scripts/ifcfg-em1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=em1
UUID=84d040c4-1070-4587-bfa8-0a67844f050e
DEVICE=em1
ONBOOT=yes
IPADDR=192.168.70.14
PREFIX=24
GATEWAY=192.168.70.1
DNS1=192.168.10.7
DOMAIN=akainix.local
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no

# cat /etc/sysconfig/network-scripts/ifcfg-em1.90
VLAN=yes
DEVICE=em1.90
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.90.14
NETMASK=255.255.255.0
PHYSDEV=em1

Now, thanks to http://www.tldp.org/HOWTO/Adv-Routing... I learned I have to do this: Everything that comes from the VLAN interface (192.168.90.14, em1.90) have to go to 192.168.90.1:

# echo 200 Testing >> /etc/iproute2/rt_tables
# ip rule add from 192.168.90.14 table Testing
# ip rule ls
0:      from all lookup local
32765:  from 192.168.90.14 lookup Testing
32766:  from all lookup main
32767:  from all lookup default

# ip route add default via 192.168.90.1 dev em1.90 table Testing
# ip route flush cache

So, Now I can reach both interfaces (em1 and em1.90) without problem: image description

So far, so good. The problem is when I try to create a second bridge… then everything goes to hell. When I have only one external bridge, this is what I configure:

# cat  ifcfg-br-ex
DEVICE="br-ex"
BOOTPROTO="static"
IPADDR="192.168.70.14"
NETMASK="255.255.255.0"
DNS1="192.168.10.7"
BROADCAST="192.168.70.255"
GATEWAY="192.168.70.1"
NM_CONTROLLED="no"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="yes"
IPV6INIT=no
ONBOOT="yes"
TYPE="OVSIntPort"
OVS_BRIDGE=br-ex
DEVICETYPE="ovs"

# cat   ifcfg-em1
DEVICE="em1"
ONBOOT="yes"
TYPE="OVSPort"
DEVICETYPE="ovs"
OVS_BRIDGE=br-ex
NM_CONTROLLED=no
IPV6INIT=no

Restart, and then start creating networks, routers, etc. in Openstack and everything goes fine since there… but now, with two external bridges I (guess) need to create… no luck. I tried stuff like http://blog.oddbit.com/2014/05/28/mul... with no luck. I also tried this:

# cat  ifcfg-br-ex90
DEVICE="br-ex90"
BOOTPROTO="static"
IPADDR="192.168.70.14"
NETMASK="255.255.255.0"
BROADCAST="192.168.70.255"
NM_CONTROLLED="no"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="yes"
IPV6INIT=no ...
(more)
edit retag flag offensive close merge delete

3 answers

Sort by » oldest newest most voted
2

answered 2016-05-02 16:09:43 -0600

sacha-m gravatar image

updated 2016-05-03 10:13:32 -0600

Nobody answer and I figured out how to do it, so I'm going to answer. Maybe is useful to somebody in the future.

I had to make a whole new approach to the problem and this was my way to do it:

Changes at OS level

I won't use VLAN at the interfaces level, but I will do it at bridge level, so em1.90 won't exists at all. so:

# cd /etc/sysconfig/network-scripts/
# cat ifcfg-em1
DEVICE="em1"
ONBOOT=yes
OVS_BRIDGE=br-salida
TYPE=OVSPort
DEVICETYPE="ovs"

# cat ifcfg-br-salida
DEVICE=br-salida
BOOTPROTO=none
ONBOOT=yes
TYPE=OVSBridge
DEVICETYPE="ovs"
IPADDR="192.168.70.14"
NETMASK="255.255.255.0"
DNS1="192.168.10.7"
GATEWAY=”192.168.70.1”

# cat ifcfg-br-salida.90
BOOTPROTO="none"
DEVICE="br-salida.90"
ONBOOT="yes"
IPADDR="192.168.90.14"
PREFIX="24"
DNS1="192.168.10.7"
VLAN=yes 
NOZEROCONF=yes
USERCTL=no

Reboot.

At this point, 192.168.70.14 should be answering, but 192.168.90.14 not. So, thanks to http://www.tldp.org/HOWTO/Adv-Routing... I did this:

We create a table called "RioBlanco" and we will configure that everything that come from 192.168.90.14 has to be assosiated to this table (rule) and goes to 192.168.90.1 (route):

# echo 200 RioBlanco >> /etc/iproute2/rt_tables
# ip rule add from 192.168.90.14 table RioBlanco
# ip rule ls
0:      from all lookup local
32765:  from 192.168.90.14 lookup RioBlanco
32766:  from all lookup main
32767:  from all lookup default
# ip route add default via 192.168.90.1 dev br-salida.90 table RioBlanco
# ip route flush cache

Now both IP are responding. This changes are not persistent. To make them persisten, we have to do this:

Rule:

# cat /etc/sysconfig/network-scripts/rule-br-salida
from 192.168.90.14 table RioBlanco

Route:

# cat /etc/sysconfig/network-scripts/route-br-salida
default via 192.168.90.1 dev br-salida.90 table RioBlanco

Changes at Openstack level

changes in /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vlan,vxlan
mechanism_drivers =openvswitch
[ml2_type_vlan]
network_vlan_ranges = fisico-vlan:90:90

Now we had to link the physical bridge (br-salida) with the interface we configured in Openstack (fisico-vlan). This has to be don in /etc/neutron/plugins/ml2/openvswitch_agent.ini:

bridge_mappings =fisico-vlan:br-salida

Now, we have to be sure of this in /etc/neutron/l3_agent.ini:

external_network_bridge =
gateway_external_network_id =

You can achieve this when you install with packstack with answer file, with this options:

CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan,vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan,vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=fisico-vlan:90:90
CONFIG_NEUTRON_L3_EXT_BRIDGE=
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=fisico-vlan:br-salida

Reboot. After the reboot, finally:

~(keystone_admin]# neutron net-create externa_vlan90 --shared --provider:network_type vlan --provider:segmentation_id 90 --provider:physical_network fisico-vlan --router:external

~(keystone_admin]# neutron subnet-create --name sub-externa_vlan90 --gateway 192.168.90.1  --allocation-pool start=192.168.90.25,end=192.168.90.35 externa_vlan90 192.168.90.0/24

After all this, Openstack is ready to work with this new network that is using VLAN tag (90), so in Horizon, we can:

  • create a ...
(more)
edit flag offensive delete link more

Comments

Please,add to your answer

ovs-ofctl dump-flows br-salida | grep NORMAL ( 5 times )
ovs-ofctl dump-flows br-int | grep dl_vlan=90
andrew.shvartz gravatar imageandrew.shvartz ( 2016-05-03 03:45:03 -0600 )edit

and one more , sorry

ip -details link show br-salida
andrew.shvartz gravatar imageandrew.shvartz ( 2016-05-03 03:55:00 -0600 )edit

Done. Why was that? Why do you need that information?

sacha-m gravatar imagesacha-m ( 2016-05-03 09:14:03 -0600 )edit

Please, add one more report

# lsmod | grep 8021
andrew.shvartz gravatar imageandrew.shvartz ( 2016-05-03 09:40:34 -0600 )edit

and one more

ip -details link show em1
andrew.shvartz gravatar imageandrew.shvartz ( 2016-05-03 10:08:06 -0600 )edit
1

answered 2016-05-05 14:00:35 -0600

dbaxps gravatar image

updated 2016-05-12 15:46:18 -0600

UPDATE 05/12/2016
@sacha-m wrote :-

Now both IP are responding. This changes are not persistent. To make them persistent, we have to do this. What means that he did a trick and both IPs 192.168.70.14 && 192.168.90.14 are responding now

Here I am loosing a point . Per ovs-ofctl dump-flows br-salida requested by Andrew Shvartz bridge br-salida became VLAN enabled after reboot and makes egress traffic vlan tagged (90).Then it should not be able any more to work as external bridge for flat network . How 192.168.70.14 can respond after conversion been done if ingress traffic is untagged. Please, clarify this moment. Via my experience if first external network is flat (vxlan) and second is vlan tagged I need two physical NICs . NIC1 supporting external flat network, NIC2 supporting external vlan network and NIC0 ( third ) for MGMT network ( em2 interface). I believe that both IPs may respond only if ingress traffic is vlan tagged 90.

END UPDATE

UPDATE 05/10/2016
What exactly I meant OVS vlan enabled bridge br-salida is not supposed to have IP at all
as parent vlan enabled device https://access.redhat.com/documentati...

yum -y install epel-release
yum -y install vconfig

# ovs-vsctl add-br br-salida
# ovs-vsctl add-port br-salida em1
# vconfig add br-salida 90

Under /etc/sysconfig/network-scripts

# cat ifcfg-em1
DEVICE="em1"
ONBOOT=yes
OVS_BRIDGE=br-salida
TYPE=OVSPort
DEVICETYPE="ovs"

# cat ifcfg-br-salida
DEVICE=br-salida
BOOTPROTO=none
ONBOOT=yes
TYPE=OVSBridge
DEVICETYPE="ovs"

# cat ifcfg-br-salida.90
BOOTPROTO="none"
DEVICE="br-salida.90"
ONBOOT="yes"
IPADDR="192.168.90.14"
PREFIX="24"
GATEWAY="192.168.90.1"
DNS1="8.8.8.8"
VLAN=yes
NOZEROCONF=yes
USERCTL=no

Then run script :-

#!/bin/bash -x
chkconfig network on
systemctl stop NetworkManager
systemctl disable NetworkManager
service network restart

END UPDATE
A quick look at https://visibilityspots.org/vlan-flat...
Clearly demonstrate, that all tricks related with "core fix"
http://www.tldp.org/HOWTO/Adv-Routing...
are completely needless in case when vlan enabled bridges are configured properly.

edit flag offensive delete link more
1

answered 2016-05-09 07:08:03 -0600

jasonwg gravatar image

Since when "Reinventing the wheel with errors" is called "A whole new approach"
See for instance
1. https://visibilityspots.org/vlan-flat...
2. http://dbaxps.blogspot.com/2015/12/ai...

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-04-27 15:14:15 -0600

Seen: 3,910 times

Last updated: May 12 '16