Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

External compute node can´t communicate with controller node

We are trying to deploy a simple system with Devstack Queens on two servers (baremetal). Our infra people have connected the two servers as displayed in the attached network diagram. (https://ibb.co/3vBkvHM) One of the servers will host the openstack controller and one compute node. The second server will be used to host another compute node. Our goal is to give the controller the possibility to choose where to deploy instances.

So far, we managed to install the controller and compute node in the first machine and compute in the second.

On the controller machine, apparently everything seems to be working. We are able to create instances, we can ping outside and are able to assign floating ips to them. Also if we create instances in the same compute node, we can ping from one to another both ways.

But then, on the other side of the coin, when we look at the second compute node, we can instantiate vms on that compute, but we can´t get them to communicate with the controller. We have noticed that if we deploy vms inside that compute, they can ping each other, but if we try to ping the gateway in that network, they can’t.

Note that when deploying compute node and discovering hosts on controller node, a patch between br-tun on both servers is created (see ovs-vsctl info appended). Though, no packet is using told path since br-tun always drops any input packet. Therefore, br-ex is the logical output preference to communicate between computes but as seen in figure, there is no connection between br-ex and Interface enp12s0.

Since we are deploying a provider network scenario, which packet flow between computes should we expect? Through br-ex(Compute External) -> Switch -> br-ex(Compute on Controller) or br-tun (Compute External) -> br-tun (Compute on Controller)

We are probably missing something somewhere. We are not trying to do anything complicated. So any help would be really appreciated.

Attached here some of the conf files:

local.conf from the controller

 [[local|localrc]]
# PASSWORDS
ADMIN_PASSWORD=*******
DATABASE_PASSWORD=supersecret
RABBIT_PASSWORD=$DATABASE_PASSWORD
SERVICE_PASSWORD=$DATABASE_PASSWORD

# HOST_IP: API endpoint host, use own public IP so endpoint can be reached from outside.
HOST_IP=XX.YY.40.124
# SERVICE_HOST: Pointer to Controller @IP
SERVICE_HOST=$HOST_IP

MULTI_HOST=1
RECLONE=True

# Logging
LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=$DEST/logs

## Neutron options
Q_USE_SECGROUP=True
PUBLIC_INTERFACE=enp132s0f1
NEUTRON_CREATE_INITIAL_NETWORKS=False
# Open vSwitch provider networking configuration
OVS_PHYSICAL_BRIDGE=br-ex

Q_ML2_TENANT_NETWORK_TYPE=vlan
Q_ML2_PLUGIN_TYPE_DRIVERS=flat,vlan,vxlan
PHYSICAL_NETWORK=default
ENABLE_TENANT_VLANS=True
TENANT_VLAN_RANGE=1200:1220

# Prevent long lockout
KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS=10
KEYSTONE_LOCKOUT_DURATION=2

# Fix incompatibility: tunnel_types starting as gre by default
[[post-config|/$Q_PLUGIN_CONF_FILE]]
[agent]
tunnel_types=vxlan

Local.conf from the compute node

[[local|localrc]]
# PASSWORDS
ADMIN_PASSWORD=********
DATABASE_PASSWORD=supersecret
RABBIT_PASSWORD=$DATABASE_PASSWORD
SERVICE_PASSWORD=$DATABASE_PASSWORD

# HOST_IP: API endpoint host, use own public IP so endpoint can be reached from outside.
HOST_IP=XX.YY.40.75
# SERVICE_HOST: Pointer to Controller @IP
SERVICE_HOST=XX.YY.40.124

MULTI_HOST=1
RECLONE=True

# Logging
LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=$DEST/logs

## Neutron options
Q_USE_SECGROUP=True
PUBLIC_INTERFACE=enp12s0
NEUTRON_CREATE_INITIAL_NETWORKS=False
# Open vSwitch provider networking configuration
OVS_PHYSICAL_BRIDGE=br-ex

Q_ML2_TENANT_NETWORK_TYPE=vlan
Q_ML2_PLUGIN_TYPE_DRIVERS=flat,vlan,vxlan
PHYSICAL_NETWORK=default
ENABLE_TENANT_VLANS=True
TENANT_VLAN_RANGE=1200:1220
ENABLED_SERVICES=n-cpu,q-meta,q-agt,q-dhcp,n-api-meta,placement-client

MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"
VNCSERVER_LISTEN=$HOST_IP
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN

# Fix incompatibility: tunnel_types starting as gre by default
[[post-config|/$Q_PLUGIN_CONF_FILE]]
[agent]
tunnel_types=vxlan

Result from ovs-vsctl show in the controller node

stack@ias4:~/devstack$ sudo ovs-vsctl show
1890df78-aeeb-4b8f-b0ff-22fb50fa69bb
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-5458284b"
            Interface "vxlan-5458284b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="XX.YY.40.124", out_key=flow, remote_ip="XX.YY.40.75"}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
        Port "enp132s0f1"
            Interface "enp132s0f1"
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "tap910b5d9d-75"
            tag: 3
            Interface "tap910b5d9d-75"
                type: internal
        Port "qvodff8c9ac-27"
            tag: 1
            Interface "qvodff8c9ac-27"
        Port "qr-8abbf087-b9"
            tag: 1
            Interface "qr-8abbf087-b9"
                type: internal
        Port "tap61cc621f-d1"
            tag: 1
            Interface "tap61cc621f-d1"
                type: internal
        Port "qvo09afe2ed-a5"
            tag: 1
            Interface "qvo09afe2ed-a5"
        Port br-int
            Interface br-int
                type: internal
        Port "qr-86e90d03-a7"
            tag: 3
            Interface "qr-86e90d03-a7"
                type: internal
        Port "qg-1a4a061c-54"
            tag: 2
            Interface "qg-1a4a061c-54"
                type: internal
        Port "qg-670a33e3-b1"
            tag: 2
            Interface "qg-670a33e3-b1"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
    ovs_version: "2.8.4"
stack@ias4:~/devstack$

Result from ovs-vsctl show on the compute node

stack@papasseit:~/devstack$ sudo ovs-vsctl show
359558d4-2e33-44de-971c-13b24dac626e
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qvo19a2581c-b4"
            tag: 1
            Interface "qvo19a2581c-b4"
        Port "tap760f1b89-1a"
            tag: 1
            Interface "tap760f1b89-1a"
                type: internal
        Port "tap3e532db9-2b"
            tag: 2
            Interface "tap3e532db9-2b"
                type: internal
        Port "qvo17e25008-ca"
            tag: 1
            Interface "qvo17e25008-ca"
        Port br-int
            Interface br-int
                type: internal
        Port "qvoce8817a8-cd"
            tag: 2
            Interface "qvoce8817a8-cd"
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-5458287c"
            Interface "vxlan-5458287c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="XX.YY.40.75", out_key=flow, remote_ip="XX.YY.40.124"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
    ovs_version: "2.8.4"
stack@papasseit:~/devstack$

result from running systemctl list-units | grep stack

stack@ias4:~/devstack$ systemctl list-units | grep stack
  devstack@c-api.service                                                                           loaded active running   Devstack devstack@c-api.service
  devstack@c-sch.service                                                                           loaded active running   Devstack devstack@c-sch.service
  devstack@c-vol.service                                                                           loaded active running   Devstack devstack@c-vol.service
  devstack@dstat.service                                                                           loaded active running   Devstack devstack@dstat.service
  devstack@etcd.service                                                                            loaded active running   Devstack devstack@etcd.service
  devstack@g-api.service                                                                           loaded active running   Devstack devstack@g-api.service
  devstack@g-reg.service                                                                           loaded active running   Devstack devstack@g-reg.service
  devstack@keystone.service                                                                        loaded active running   Devstack devstack@keystone.service
  devstack@n-api-meta.service                                                                      loaded active running   Devstack devstack@n-api-meta.service
  devstack@n-api.service                                                                           loaded active running   Devstack devstack@n-api.service
  devstack@n-cauth.service                                                                         loaded active running   Devstack devstack@n-cauth.service
  devstack@n-cond-cell1.service                                                                    loaded active running   Devstack devstack@n-cond-cell1.service
  devstack@n-cpu.service                                                                           loaded active running   Devstack devstack@n-cpu.service
  devstack@n-novnc.service                                                                         loaded active running   Devstack devstack@n-novnc.service
  devstack@n-sch.service                                                                           loaded active running   Devstack devstack@n-sch.service
  devstack@n-super-cond.service                                                                    loaded active running   Devstack devstack@n-super-cond.service
  devstack@placement-api.service                                                                   loaded active running   Devstack devstack@placement-api.service
  devstack@q-agt.service                                                                           loaded active running   Devstack devstack@q-agt.service
  devstack@q-dhcp.service                                                                          loaded active running   Devstack devstack@q-dhcp.service
  devstack@q-l3.service                                                                            loaded active running   Devstack devstack@q-l3.service
  devstack@q-meta.service                                                                          loaded active running   Devstack devstack@q-meta.service
  devstack@q-svc.service                                                                           loaded active running   Devstack devstack@q-svc.service
  system-devstack.slice                                                                            loaded active active    system-devstack.slice
stack@ias4:~/devstack$

And last but not least results from running systemctl list-units | grep stack on the compute node

stack@papasseit:~/devstack$ systemctl list-units | grep stack
● devstack@c-sch.service                                                                                not-found failed failed    devstack@c-sch.service
● devstack@c-vol.service                                                                                not-found failed failed    devstack@c-vol.service
  devstack@n-api-meta.service                                                                           loaded    active running   Devstack devstack@n-api-meta.service
  devstack@n-cpu.service                                                                                loaded    active running   Devstack devstack@n-cpu.service
● devstack@n-sch.service                                                                                not-found failed failed    devstack@n-sch.service
  devstack@q-agt.service                                                                                loaded    active running   Devstack devstack@q-agt.service
  devstack@q-dhcp.service                                                                               loaded    active running   Devstack devstack@q-dhcp.service
  devstack@q-meta.service                                                                               loaded    active running   Devstack devstack@q-meta.service
● devstack@q-svc.service                                                                                not-found failed failed    devstack@q-svc.service
  system-devstack.slice                                                                                 loaded    active active    system-devstack.slice
stack@papasseit:~/devstack$