Ask Your Question

Mapping physical and virtual networking

asked 2019-10-03 15:53:43 -0500

awalters gravatar image

updated 2019-10-04 15:50:04 -0500

I'm trying to understand the best way to set up physical networking for OpenStack, and having some troubles getting my head around how to navigate the possible options. I'm just working with a single node at the moment, so not even worrying about communication between nodes yet.

So, let's assume this is running on CentOS, although obviously that shouldn't matter much to the answers, but it's easiest to call it something. For the VMs, I want them to be able to access the internet over, let's say vlan 20 (and let's say it's 192.168.1.x), and an office network on vlan 30 (10.10.10.x). There is also a server/management vlan for the office, 40 (172.16.1.x).

  • On the CentOS box, I'd assume it generally should have just one IP, in the management vlan (172.16.1.x), right?
  • Would the physical networking to the box generally be a single (presumably teamed) trunk port? Or one for each vlan (so in this case 3)?
  • If it's a trunk port, how does the CentOS box itself get its traffic tagged, so it can communicate?
  • Or if it's one for each vlan, how does that work for the gateway on CentOS, if these vlans maybe don't all have routing between them?
  • For teaming, is it best to do that in linux or with ovs tools? (Adding bonds using ovs-ofctl, or...?)

I've tried various permutations of all of these and have had various issues. I want to concentrate on solving the ones that lie in the way of my best path, not just every possibility I come across.

ETA: To further elaborate on questions 1 and 3: currently the CentOS box has an IP on the management vlan (40). I also want one of the VMs to be able to communication with this vlan. However, it is currently not working. Other external network communication is working, but not this one.

When I examined the traffic, I discovered that the issue is this - when a request from a VM goes out on vlan 20 (well, whatever OpenStack's internal id is, which the a flow in br-ex then mods to 20), I can see the packet on br-ex, then on the physical port, with vlan 20. The reply then comes back from the switch to the physical port on vlan 20, then to br-ex, then to br-int which flips it back to the internal id. This is all fine. But with vlan 40, since CentOS is on it as well, there's the extra complication of tagging on the CentOS packets as well.

When I first installed CentOS I was trying to do this with a sub-interface. This doesn't work with OpenStack since then the vlan on the subinterface doesn't match some of the VM traffic. So I then set the trunk port on the switch to have the same native vlan as ... (more)

edit retag flag offensive close merge delete


A production system tends to use a single NIC bond and implement all the networks as VLANs. Linux has a VLAN NIC type that strips VLAN tags from incoming packets and adds them to outgoing packets.

For an example of such a setup, see (old but still relevant).

Bernd Bausch gravatar imageBernd Bausch ( 2019-10-03 23:26:41 -0500 )edit

Hmmm....I saw that in my searches, but since it explicitly states that it's not how the manuals say to do it multiple times, I wondered how common it was. :-) And also how relevant, because of the age (as you mention). Thanks! I don't think it answers all my questions, but certainly some.

awalters gravatar imageawalters ( 2019-10-04 11:49:39 -0500 )edit

So I've got an answer to the second question - it's a trunk port. That makes 4 irrelevant. Not seeing anything in that guide about how to implement teaming/bonding, though. Also not clear (though I'll be going through it more) about what network the CentOS install itself has an IP on, and tagging

awalters gravatar imageawalters ( 2019-10-04 12:09:52 -0500 )edit

re: bonding, seeing in a RHEL guide a suggestion to use OVS bonding for networks that interact with OS VMs, and Linux bonding for control or storage networks "because OVS bonds carry the potential for control plane disruption that can occur when OVS or the neutron agent". Seems to make sense...

awalters gravatar imageawalters ( 2019-10-04 12:12:35 -0500 )edit

I have done some work with the Suse OpenStack cloud. Up to a year ago, it used Linux bonding.

Bernd Bausch gravatar imageBernd Bausch ( 2019-10-04 17:07:14 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2019-10-31 21:00:34 -0500

awalters gravatar image

I was able to do this with a trunk port and a single (management) IP on the CentOS box by creating two OpenStack bridges - I made br-ex for all networks other than the management network, set all of the associated OS networks up as VLAN, and attached br-ex to the physical port/team. I then made br-mgmt, put the management network on that as a flat network, and attached it to a tagged subinterface of the physical port/team. I gave br-mgmt the IP for the box, and didn't put an IP at all for br-ex. Seems to work fine, even if it was a pain to figure out. I'm curious if this is how people generally solve this (cause I didn't find instructions for this anywhere), or what else people do instead, but...what I need is up and working, anyway.

As mentioned above in the comments, I made the team in Linux rather than OVS due to a recommendation from the RHEL guide.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2019-10-03 15:53:43 -0500

Seen: 223 times

Last updated: Oct 31 '19