Mapping physical and virtual networking

asked 2019-10-03 15:53:43 -0500

allisonw gravatar image

updated 2019-10-04 15:50:04 -0500

I'm trying to understand the best way to set up physical networking for OpenStack, and having some troubles getting my head around how to navigate the possible options. I'm just working with a single node at the moment, so not even worrying about communication between nodes yet.

So, let's assume this is running on CentOS, although obviously that shouldn't matter much to the answers, but it's easiest to call it something. For the VMs, I want them to be able to access the internet over, let's say vlan 20 (and let's say it's 192.168.1.x), and an office network on vlan 30 (10.10.10.x). There is also a server/management vlan for the office, 40 (172.16.1.x).

  • On the CentOS box, I'd assume it generally should have just one IP, in the management vlan (172.16.1.x), right?
  • Would the physical networking to the box generally be a single (presumably teamed) trunk port? Or one for each vlan (so in this case 3)?
  • If it's a trunk port, how does the CentOS box itself get its traffic tagged, so it can communicate?
  • Or if it's one for each vlan, how does that work for the gateway on CentOS, if these vlans maybe don't all have routing between them?
  • For teaming, is it best to do that in linux or with ovs tools? (Adding bonds using ovs-ofctl, or...?)

I've tried various permutations of all of these and have had various issues. I want to concentrate on solving the ones that lie in the way of my best path, not just every possibility I come across.

ETA: To further elaborate on questions 1 and 3: currently the CentOS box has an IP on the management vlan (40). I also want one of the VMs to be able to communication with this vlan. However, it is currently not working. Other external network communication is working, but not this one.

When I examined the traffic, I discovered that the issue is this - when a request from a VM goes out on vlan 20 (well, whatever OpenStack's internal id is, which the a flow in br-ex then mods to 20), I can see the packet on br-ex, then on the physical port, with vlan 20. The reply then comes back from the switch to the physical port on vlan 20, then to br-ex, then to br-int which flips it back to the internal id. This is all fine. But with vlan 40, since CentOS is on it as well, there's the extra complication of tagging on the CentOS packets as well.

When I first installed CentOS I was trying to do this with a sub-interface. This doesn't work with OpenStack since then the vlan on the subinterface doesn't match some of the VM traffic. So I then set the trunk port on the switch to have the same native vlan as ... (more)

edit retag flag offensive close merge delete


A production system tends to use a single NIC bond and implement all the networks as VLANs. Linux has a VLAN NIC type that strips VLAN tags from incoming packets and adds them to outgoing packets.

For an example of such a setup, see (old but still relevant).

Bernd Bausch gravatar imageBernd Bausch ( 2019-10-03 23:26:41 -0500 )edit

Hmmm....I saw that in my searches, but since it explicitly states that it's not how the manuals say to do it multiple times, I wondered how common it was. :-) And also how relevant, because of the age (as you mention). Thanks! I don't think it answers all my questions, but certainly some.

allisonw gravatar imageallisonw ( 2019-10-04 11:49:39 -0500 )edit

So I've got an answer to the second question - it's a trunk port. That makes 4 irrelevant. Not seeing anything in that guide about how to implement teaming/bonding, though. Also not clear (though I'll be going through it more) about what network the CentOS install itself has an IP on, and tagging

allisonw gravatar imageallisonw ( 2019-10-04 12:09:52 -0500 )edit

re: bonding, seeing in a RHEL guide a suggestion to use OVS bonding for networks that interact with OS VMs, and Linux bonding for control or storage networks "because OVS bonds carry the potential for control plane disruption that can occur when OVS or the neutron agent". Seems to make sense...

allisonw gravatar imageallisonw ( 2019-10-04 12:12:35 -0500 )edit

I have done some work with the Suse OpenStack cloud. Up to a year ago, it used Linux bonding.

Bernd Bausch gravatar imageBernd Bausch ( 2019-10-04 17:07:14 -0500 )edit