Nested virtualization with KVM/OpenVSwitch

asked 2017-05-04 01:50:26 -0600

Micha gravatar image

updated 2017-05-10 04:07:31 -0600

Hi all together,

I have a OpenStack Installation running on an bunch of old hardware we put together from old PCs. Now I want to set up OpenStack on a single host, but inside multiple VMs virtualized with KVM. The KVMs are connected via OpenVSwitch. The network setup is the following: (Openstack Network Setup)

I can successfully ping each instance over the Management network, running over intbr0. I can also ping the compute1, compute2 and network via their IP Addresses on the second NIC, which is connected to the tunbr0. A third NIC on the network node is set to 'manual' and attached to the br-ext. I then set up keystone, glance, neutron and nova. Inside the VMs I use OpenVSwitch again, which is this time managed by OpenStack, for the inter-vm traffic I use vxlan tunnels. The connection to the outer world is done via VLAN, where I implemented trunking on both OpenVSwitches (br-ext and databr0)

I was able to create multiple instances which appear correctly on the compute nodes and are shown without an error. I also created two networks - one internal Test-Customer network and one external network. I also created the needed subnets and the router and attached it to both networks correctly.

After doing so I was able to: Ping the Router IPs from Routers Namespace (Internal IP and gateway IP)

I was NOT able to: Ping the attached VMs Ping another machine in the same Subnet attached to databr0 Ping the Router Gateway from the machine attached to databr0

However, my instances got IPs from the correct subnet after booting up, so I assume some DHCP magic was working (but why is my ping failing then?).

I had a setup like that running on ESXI. I remember I had to set the vSwitch to promiscous mode. However, I did nothing like that on databr0, tunbr0 or intbr0 which is - as I assume - the reason packets may get dropped there. I found (THIS) article, however I'm not sure how to set that up.

If anyone could help me to get this working, I would highly appreciate it. OpenVSwitch and OpenFlow looks like some dark art to me right now and I'm stuck at this point for over a week. If you need more information, please feel free to ask :)

Greetings from Germany Micha

EDIT: More infos: Oh, just remembered I can't be sure the instance got the IP - I was just checking the metadata (which may be incorrect).

The network is not configured in virsh (yet - should I do that?), I attached the instances by editing their config file and adding another interface like this:

<interface type='bridge'>   
  <mac address='52:54:00:43:68:01'/>  
  <source bridge='ostun0'/>  
  <virtualport type='openvswitch'>
    <parameters interfaceid='85dc9976-0f7b-4995-891d-c1ce2f5902a6'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> 

However, I also found out that virsh apparently does ... (more)

edit retag flag offensive close merge delete


Could you check the security group rules? Allow icmp must be set. By default this is disabled.

sanjana gravatar imagesanjana ( 2017-05-04 23:30:02 -0600 )edit

Since the router resides (afaik) in the same security group as the node, it should be pingable by the default mitaka rules.

This would also not explain why I can't ping the external public IPs.

Can somebody tell me if the IP after boot is assigned via DHCP or if there is some voodoo happening?

Micha gravatar imageMicha ( 2017-05-05 04:12:26 -0600 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2017-05-07 12:36:25 -0600

mlschuler gravatar image

An instance usually gets its IP address assigned by DHCP. In your installation, what is the configuration of the network within KVM? Would you show us the output of virsh net-dumpxml?

edit flag offensive delete link more


Added info to the start post since there is no 'reply' function here.

Micha gravatar imageMicha ( 2017-05-10 04:07:06 -0600 )edit

I have done a similar installation for a 'virtualized' SUSE OpenStack Cloud. The difference to what I see in your setting, is that I used portgroups in openvswitch for each of the node types in OpenStack (compute, controller, storage etc). I do not see any VLANs in your xml.

mlschuler gravatar imagemlschuler ( 2017-05-10 04:21:07 -0600 )edit

I added the VLANs via ovs-vsctl set port vnet8 trunks=2003,2004

Micha gravatar imageMicha ( 2017-05-10 05:16:03 -0600 )edit

I do not understand why dev dev1 on databr0 is not a VLAN trunk with IDs 2003 and 2004? The traffic from vnet8 does not go to anywhere.

mlschuler gravatar imagemlschuler ( 2017-05-10 09:50:58 -0600 )edit

Let me repeat my previous question in another way. It does not make sense to define a single switch port (vnet8) as a VLAN trunk and all other ports as native VLANs. 2003 and 2004 need to be defined for other ports too. Did I miss something?

mlschuler gravatar imagemlschuler ( 2017-05-11 10:42:26 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2017-05-04 01:50:26 -0600

Seen: 813 times

Last updated: May 10 '17