Ask Your Question

Revision history [back]

DHCP in kolla-ansible / VMWare / DVS setting

Dear community,

currently, we are in the process of deploying OpenStack Stein for evaluation purposes. Our target platform (hypervisor) is a VMWare environment in version 6.0.0 (only targeting provider networks without NSX for now). OpenStack nodes are deployed as VMWare VMs in one cluster (os: Ubuntu), tenant VMs should be spawned in a second cluster. We deploy OpenStack via kolla-ansible (8.0.0.0rc1) and followed a default multinode inventory with three control and two network nodes.

Our main problem is that VMs (cirros) can be spawned in a provider network created via OpenStack (dhcp enabled), however, they don't get an ip configuration via dhcp. Now, we are unsure how to debug our dhcp issues and would welcome any tips.

Many thanks for any help and best regards

David


Some more background on our current setup and performed tasks:

According to the kolla-ansible installation guide, each OpenStack node has two NICs for the 'network_interface' and 'neutron_external_interface' that attach to corresponding port groups on a VMWare distributed virtual switch 'OS-DVS'. All ESX hosts are attached to this DVS via physical NICs. The 'neutron_external_interface' port group has 'forged transmits' enabled.

We followed the following instructions on Neutron Networking with VMWare VSphere: https://docs.openstack.org/nova/stein/admin/configuration/hypervisor-vmware.html#networking-with-vmware-vsphere

If you are using the OpenStack Networking Service: Before provisioning VMs, create a port group with the same name as the vmware.integration_bridge value in nova.conf (default is br-int). All VM NICs are attached to this port group for management by the OpenStack Networking plug-in.

But with the naming kolla-ansible demands, i.e., we created a port group named 'br-dvs' on the VMWare DVS which has forged transmits enabled: https://docs.openstack.org/kolla-ansible/queens/reference/vmware-guide.html#vmware-nsx-dvs

For VMware DVS, the Neutron DHCP agent does not attaches to Open vSwitch inside VMware environment, but attach to the Open vSwitch bridge called br-dvs on the OpenStack side and replies to/receives DHCP packets through VLAN. Similar to what the DHCP agent does, Neutron metadata agent attaches to br-dvs bridge and works through VLAN.

The following snippet shows the network configuration of one of the network nodes. ens160 is the 'network_interface', ens192 is the 'neutron_external_interface', and ens224 corresponds to the VMWare port group 'br-dvs'.

openstack@network01:~$ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b4:87:05 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.21/24 brd 10.10.10.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:8705/64 scope link
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master ovs-system state DOWN group default qlen 1000
    link/ether 00:50:56:b4:bb:73 brd ff:ff:ff:ff:ff:ff
4: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:50:56:b4:6b:b7 brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:fc:51:d4:fd brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2e:d6:62:c6:42:2b brd ff:ff:ff:ff:ff:ff
7: br-dvs: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:50:56:b4:bb:73 brd ff:ff:ff:ff:ff:ff

The following snippet shows the output of ovs-vsctl on one of the network nodes:

openstack@network01:~$ sudo ovs-vsctl show
02c90c70-3cdb-4830-8c90-2669fe194f7b
    ovs_version: "2.9.2"
openstack@network01:~$

I believe in the neutron-dhcp-agent logs I do not see any dhcp requests. However, some configuration actions seem to happen for the spawned vm:

Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=compute01, created_at=2019-05-14T13:50:12Z, description=, device_id=bba01edb-c8e2-4235-8c09-a5ff059efda3, device_owner=compute:nova, fixed_ips=[{u'subnet_id': u'5aba6f74-aabf-41c1-9c63-7e646a59993c', u'ip_address': u'10.10.11.111'}], id=fc658fbc-9db9-4be6-b720-5afb320ab5f2, mac_address=fa:16:3e:7d:d5:4a, name=, network_id=380cde0b-cd3b-4441-bc3a-12e2587cb0e7, port_security_enabled=False, project_id=8a3f53fe893f491f8c9065ac6318f3d1, revision_number=13, security_groups=[], status=ACTIVE, tags=[], tenant_id=8a3f53fe893f491f8c9065ac6318f3d1, updated_at=2019-05-14T13:55:09Z

Are there any additional requirements on the hardware side? Right now we are using a cheapo 5-port switch for testing purposes that probably does not recognize VLAN ids.

Also, my ovs knowledge is still rather limit and any tips on troubleshooting it are appreciated. Please let me know if you need to see any additional configuration/logs.

DHCP in kolla-ansible / VMWare / DVS setting

Update 2019-05-15:

Some progress I believe. However, I still don't see dhcp requests from my test vm arriving at the OpenStack network nodes. What I do not quite understand: When I create a flat network in OpenStack a new VMWare port group on the VMWare distributed switch is created and the vm attaches to it. However:

  • First, the vm does not attached to any other OpenStack control network / VMWare port group (at least I don't see an appropriate nic)
  • Second, apparently none of the OpenStack network nodes attaches to the created flat network / VMWare port group.

Hence, I don't understand how vm dhcp requests are supposed to reach the network nodes in the first place. Do I have to create an appropriate nic on the vm or on the network nodes by myself?

Current status: I did notice that on my network nodes some deployment issues regarding the network interfaces must have happended since the ovs db was empty. A reboot of the network notes and re-deployment of OpenStack did seem to help.

Current ovs db:

root@network01:~# ovs-vsctl show
7465c5b6-6e70-43c7-843d-522214340362
    Bridge br-dvs
        Port br-dvs
            Interface br-dvs
                type: internal
        Port "tapde20750b-c2"
            tag: 4095
            Interface "tapde20750b-c2"
                type: internal
        Port "ens192"
            Interface "ens192"

Then, I saw that the neutron-dhcp-agent tried to apply some iptables rules. Initially that failed but I could fix it by loading the ip6-tables kernel module manually on the network nodes. However, when listing all iptables rules I cannot find any neutron-specific rules - Maybe this is an issue right now?

iptables:

root@network01:~# sudo iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

No dhcp request at network nodes:

root@network01:~# cat /var/log/syslog | grep DHCP
root@network01:~#

Dear community,

currently, we are in the process of deploying OpenStack Stein for evaluation purposes. Our target platform (hypervisor) is a VMWare environment in version 6.0.0 (only targeting provider networks without NSX for now). OpenStack nodes are deployed as VMWare VMs in one cluster (os: Ubuntu), tenant VMs should be spawned in a second cluster. We deploy OpenStack via kolla-ansible (8.0.0.0rc1) and followed a default multinode inventory with three control and two network nodes.

Our main problem is that VMs (cirros) can be spawned in a provider network created via OpenStack (dhcp enabled), however, they don't get an ip configuration via dhcp. Now, we are unsure how to debug our dhcp issues and would welcome any tips.

Many thanks for any help and best regards

David


Some more background on our current setup and performed tasks:

According to the kolla-ansible installation guide, each OpenStack node has two NICs for the 'network_interface' and 'neutron_external_interface' that attach to corresponding port groups on a VMWare distributed virtual switch 'OS-DVS'. All ESX hosts are attached to this DVS via physical NICs. The 'neutron_external_interface' port group has 'forged transmits' enabled.

We followed the following instructions on Neutron Networking with VMWare VSphere: https://docs.openstack.org/nova/stein/admin/configuration/hypervisor-vmware.html#networking-with-vmware-vsphere

If you are using the OpenStack Networking Service: Before provisioning VMs, create a port group with the same name as the vmware.integration_bridge value in nova.conf (default is br-int). All VM NICs are attached to this port group for management by the OpenStack Networking plug-in.

But with the naming kolla-ansible demands, i.e., we created a port group named 'br-dvs' on the VMWare DVS which has forged transmits enabled: https://docs.openstack.org/kolla-ansible/queens/reference/vmware-guide.html#vmware-nsx-dvs

For VMware DVS, the Neutron DHCP agent does not attaches to Open vSwitch inside VMware environment, but attach to the Open vSwitch bridge called br-dvs on the OpenStack side and replies to/receives DHCP packets through VLAN. Similar to what the DHCP agent does, Neutron metadata agent attaches to br-dvs bridge and works through VLAN.

The following snippet shows the network configuration of one of the network nodes. ens160 is the 'network_interface', ens192 is the 'neutron_external_interface', and ens224 corresponds to the VMWare port group 'br-dvs'.

openstack@network01:~$ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b4:87:05 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.21/24 brd 10.10.10.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:8705/64 scope link
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master ovs-system state DOWN group default qlen 1000
    link/ether 00:50:56:b4:bb:73 brd ff:ff:ff:ff:ff:ff
4: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:50:56:b4:6b:b7 brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:fc:51:d4:fd brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2e:d6:62:c6:42:2b brd ff:ff:ff:ff:ff:ff
7: br-dvs: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:50:56:b4:bb:73 brd ff:ff:ff:ff:ff:ff

The following snippet shows the output of ovs-vsctl on one of the network nodes:

openstack@network01:~$ sudo ovs-vsctl show
02c90c70-3cdb-4830-8c90-2669fe194f7b
    ovs_version: "2.9.2"
openstack@network01:~$

I believe in the neutron-dhcp-agent logs I do not see any dhcp requests. However, some configuration actions seem to happen for the spawned vm:

Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=compute01, created_at=2019-05-14T13:50:12Z, description=, device_id=bba01edb-c8e2-4235-8c09-a5ff059efda3, device_owner=compute:nova, fixed_ips=[{u'subnet_id': u'5aba6f74-aabf-41c1-9c63-7e646a59993c', u'ip_address': u'10.10.11.111'}], id=fc658fbc-9db9-4be6-b720-5afb320ab5f2, mac_address=fa:16:3e:7d:d5:4a, name=, network_id=380cde0b-cd3b-4441-bc3a-12e2587cb0e7, port_security_enabled=False, project_id=8a3f53fe893f491f8c9065ac6318f3d1, revision_number=13, security_groups=[], status=ACTIVE, tags=[], tenant_id=8a3f53fe893f491f8c9065ac6318f3d1, updated_at=2019-05-14T13:55:09Z

Are there any additional requirements on the hardware side? Right now we are using a cheapo 5-port switch for testing purposes that probably does not recognize VLAN ids.

Also, my ovs knowledge is still rather limit and any tips on troubleshooting it are appreciated. Please let me know if you need to see any additional configuration/logs.