Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Single interface multinode

Hi everyone I was hoping to get some help if anyone has some time Ive run into a real conundrum!

I have 3 Intel NUCs each with one interface (eno1). Im trying to get three node setup (controller, network, compute) and I am having diffuculty with the external network since I am VLAN'ing to create the three networks needed. It "seems" like everything else is working - here is my configuration.

Controller node: VLAN 5 10.7.5.11

Compute: VLAN 5 10.7.5.31

VLAN 6 10.7.6.31

Network: VLAN 5 10.7.5.21

VLAN 6 10.7.6.21

VLAN 10 This is the unnumbered interface used for the external network

The network gateway is 10.7.0.1

So basically what is happening, I can create two instances on the tenant network (192.168.1.2 and .4) and they are able to ping each other and the tenant gateway (192.168.1.4). Ive assigned floating IP address' (10.7.5.102 and 103) and they are able to ping each other on the floating ip address but not out to any other host on the 10.7.5.x network nor out to the internet. I believe this is beacuse the br-ex is not working correctly (maybe because of the VLAN on the single interface?)

Here are some more configurations

Network: [root@network ~]# ovs-vsctl show 4942d9b6-0406-4d87-80f7-a7de63d6ac85 Bridge br-tun Port "gre-0a07061f" Interface "gre-0a07061f" type: gre options: {in_key=flow, local_ip="10.7.6.21", out_key=flow, remote_ip="10.7.6.31"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "tap596ffb99-96" tag: 1 Interface "tap596ffb99-96" type: internal Port br-int Interface br-int type: internal Port "qr-f77d4953-1a" tag: 1 Interface "qr-f77d4953-1a" type: internal Bridge br-ex Port "eno1.10" Interface "eno1.10" Port br-ex Interface br-ex type: internal Port "qg-8a50284b-db" Interface "qg-8a50284b-db" type: internal ovs_version: "2.0.0"

[root@network ~]# ip netns list qdhcp-a94024e7-6749-481b-a8e6-511c95fb22e6 qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f

[root@network ~]# ip netns exec qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.7.5.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-8a50284b-db 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-f77d4953-1a

[root@network ~]# ip netns exec qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f ip a 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 5: qg-8a50284b-db: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:2b:dc:0d brd ff:ff:ff:ff:ff:ff inet 10.7.5.101/24 brd 10.7.5.255 scope global qg-8a50284b-db valid_lft forever preferred_lft forever inet 10.7.5.102/32 brd 10.7.5.102 scope global qg-8a50284b-db valid_lft forever preferred_lft forever inet 10.7.5.103/32 brd 10.7.5.103 scope global qg-8a50284b-db valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe2b:dc0d/64 scope link valid_lft forever preferred_lft forever 6: qr-f77d4953-1a: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ca:d2:75 brd ff:ff:ff:ff:ff:ff inet 192.168.1.1/24 brd 192.168.1.255 scope global qr-f77d4953-1a valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feca:d275/64 scope link valid_lft forever preferred_lft forever

[root@network ~]# neutron net-list +--------------------------------------+----------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+-----------------------------------------------------+ | a94024e7-6749-481b-a8e6-511c95fb22e6 | demo-net | 8d0cdad1-a27a-4ede-8442-5bbb0d0eeb57 192.168.1.0/24 | | d92791d7-0c24-45e4-9019-bbf73dae3e74 | ext-net | c8427da2-3eca-4cf0-bdc9-84e2aaccbbe8 10.7.5.0/24 | +--------------------------------------+----------+-----------------------------------------------------+

[root@network ~]# neutron subnet-list +--------------------------------------+-------------+----------------

+--------------------------------------------------+ | id | name | cidr | allocation_pools

 |

+--------------------------------------+-------------+----------------

+--------------------------------------------------+ | 8d0cdad1-a27a-4ede-8442-5bbb0d0eeb57 | demo-subnet | 192.168.1.0/24 | {"start": "192.168.1.2", "end":

"192.168.1.254"} | | c8427da2-3eca-4cf0-bdc9-84e2aaccbbe8 | ext-subnet | 10.7.5.0/24 | {"start": "10.7.5.101", "end":

"10.7.5.200"} | +--------------------------------------+-------------+----------------

+--------------------------------------------------+

Compute:

[root@compute1 ~]# ovs-vsctl show 85d2b96c-5290-4d2e-b623-502725d2b68c Bridge br-tun Port br-tun Interface br-tun type: internal Port "gre-0a070615" Interface "gre-0a070615" type: gre options: {in_key=flow, local_ip="10.7.6.31", out_key=flow, remote_ip="10.7.6.21"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvobc53a4f9-86" tag: 1 Interface "qvobc53a4f9-86" Port "qvoe3a873c9-ea" tag: 1 Interface "qvoe3a873c9-ea" ovs_version: "2.0.0"

Another wierd thing I noticed is that from the instance when I ping either the netowrk node or the the compute node I will get see the pings on the network and compute node but I will not see any response on the instance doing the ping!

From instance to network node: Instance:

ping 10.7.5.21

PING 10.7.5.21 (10.7.5.21): 56 data bytes 3 packets transmitted, 0 packets recieved, 100% packet loss

Network (notice the IP, it has the external IP - on compute node itll see it as the tenant IP!): [root@network ~]# tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo' tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes 11:09:07.570023 IP (tos 0x0, ttl 63, id 50968, offset 0, flags [DF], proto ICMP (1), length 84) 10.7.5.102 > 10.7.5.21: ICMP echo request, id 20481, seq 0, length 64 11:09:07.570032 ethertype IPv4, IP (tos 0x0, ttl 63, id 50968, offset 0, flags [DF], proto ICMP (1), length 84) 10.7.5.102 > 10.7.5.21: ICMP echo request, id 20481, seq 0, length 64 11:09:07.570021 IP (tos 0x0, ttl 63, id 50968, offset 0, flags [DF], proto ICMP (1), length 84) 10.7.5.102 > 10.7.5.21: ICMP echo request, id 20481, seq 0, length 64

From instance to compute node (notice the tenant IP):

Instance:

ping 10.7.5.31

PING 10.7.5.31 (10.7.5.31): 56 data bytes 3 packets transmitted, 0 packets recieved, 100% packet loss

Compute1: [root@compute1 ~]# tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo' tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

11:13:04.765748 IP (tos 0x0, ttl 64, id 43692, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.1.2 > 10.7.5.31: ICMP echo request, id 21249, seq 0, length 64 11:13:04.765780 IP (tos 0x0, ttl 64, id 43692, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.1.2 > 10.7.5.31: ICMP echo request, id 21249, seq 0, length 64 11:13:04.765782 IP (tos 0x0, ttl 64, id 43692, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.1.2 > 10.7.5.31: ICMP echo request, id 21249, seq 0, length 64

Pings to the controller node from the instances are not seen.

click to hide/show revision 2
No.2 Revision

Single interface multinode

Hi everyone I was hoping to get some help if anyone has some time Ive run into a real conundrum!

I have 3 Intel NUCs each with one interface (eno1). Im trying to get three node setup (controller, network, compute) and I am having diffuculty with the external network since I am VLAN'ing to create the three networks needed. It "seems" like everything else is working - here is my configuration.

Controller node: VLAN 5 10.7.5.11

Compute: VLAN 5 10.7.5.31

VLAN 6 10.7.6.31

Network: VLAN 5 10.7.5.21

VLAN 6 10.7.6.21

VLAN 10 This is the unnumbered interface used for the external network

The network gateway is 10.7.0.1

So basically what is happening, I can create two instances on the tenant network (192.168.1.2 and .4) and they are able to ping each other and the tenant gateway (192.168.1.4). Ive assigned floating IP address' (10.7.5.102 and 103) and they are able to ping each other on the floating ip address but not out to any other host on the 10.7.5.x network nor out to the internet. I believe this is beacuse the br-ex is not working correctly (maybe because of the VLAN on the single interface?)

Here are some more configurations

Network: Network:

[root@network ~]#  ovs-vsctl show
4942d9b6-0406-4d87-80f7-a7de63d6ac85
    Bridge br-tun
        Port "gre-0a07061f"
            Interface "gre-0a07061f"
                type: gre
                options: {in_key=flow, local_ip="10.7.6.21", out_key=flow, remote_ip="10.7.6.31"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap596ffb99-96"
            tag: 1
            Interface "tap596ffb99-96"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "qr-f77d4953-1a"
            tag: 1
            Interface "qr-f77d4953-1a"
                type: internal
    Bridge br-ex
        Port "eno1.10"
            Interface "eno1.10"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-8a50284b-db"
            Interface "qg-8a50284b-db"
                type: internal
    ovs_version: "2.0.0"

"2.0.0" [root@network ~]# ip netns list qdhcp-a94024e7-6749-481b-a8e6-511c95fb22e6 qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f

qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f [root@network ~]# ip netns exec qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.7.5.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-8a50284b-db 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-f77d4953-1a

qr-f77d4953-1a [root@network ~]# ip netns exec qrouter-8a29f0cf-34ef-42bb-a16f-d849402b386f ip a 1: lo: <loopback,up,lower_up> <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 5: qg-8a50284b-db: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:2b:dc:0d brd ff:ff:ff:ff:ff:ff inet 10.7.5.101/24 brd 10.7.5.255 scope global qg-8a50284b-db valid_lft forever preferred_lft forever inet 10.7.5.102/32 brd 10.7.5.102 scope global qg-8a50284b-db valid_lft forever preferred_lft forever inet 10.7.5.103/32 brd 10.7.5.103 scope global qg-8a50284b-db valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe2b:dc0d/64 scope link valid_lft forever preferred_lft forever 6: qr-f77d4953-1a: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ca:d2:75 brd ff:ff:ff:ff:ff:ff inet 192.168.1.1/24 brd 192.168.1.255 scope global qr-f77d4953-1a valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feca:d275/64 scope link valid_lft forever preferred_lft forever

forever [root@network ~]# neutron net-list +--------------------------------------+----------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+-----------------------------------------------------+ | a94024e7-6749-481b-a8e6-511c95fb22e6 | demo-net | 8d0cdad1-a27a-4ede-8442-5bbb0d0eeb57 192.168.1.0/24 | | d92791d7-0c24-45e4-9019-bbf73dae3e74 | ext-net | c8427da2-3eca-4cf0-bdc9-84e2aaccbbe8 10.7.5.0/24 | +--------------------------------------+----------+-----------------------------------------------------+

+--------------------------------------+----------+-----------------------------------------------------+ [root@network ~]# neutron subnet-list +--------------------------------------+-------------+----------------

+--------------------------------------+-------------+---------------- +--------------------------------------------------+ | id | name | cidr | allocation_pools

  |

+--------------------------------------+-------------+----------------

+--------------------------------------+-------------+---------------- +--------------------------------------------------+ | 8d0cdad1-a27a-4ede-8442-5bbb0d0eeb57 | demo-subnet | 192.168.1.0/24 | {"start": "192.168.1.2", "end":

"192.168.1.254"} | | c8427da2-3eca-4cf0-bdc9-84e2aaccbbe8 | ext-subnet | 10.7.5.0/24 | {"start": "10.7.5.101", "end":

"10.7.5.200"} | +--------------------------------------+-------------+----------------

+--------------------------------------------------+

+--------------------------------------+-------------+---------------- +--------------------------------------------------+

Compute:

[root@compute1 ~]# ovs-vsctl show
85d2b96c-5290-4d2e-b623-502725d2b68c
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0a070615"
            Interface "gre-0a070615"
                type: gre
                options: {in_key=flow, local_ip="10.7.6.31", out_key=flow, remote_ip="10.7.6.21"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "qvobc53a4f9-86"
            tag: 1
            Interface "qvobc53a4f9-86"
        Port "qvoe3a873c9-ea"
            tag: 1
            Interface "qvoe3a873c9-ea"
    ovs_version: "2.0.0"

"2.0.0"

Another wierd thing I noticed is that from the instance when I ping either the netowrk node or the the compute node I will get see the pings on the network and compute node but I will not see any response on the instance doing the ping!

From instance to network node: Instance:

ping 10.7.5.21

#ping 10.7.5.21
PING 10.7.5.21 (10.7.5.21): 56 data bytes
3 packets transmitted, 0 packets recieved, 100% packet loss

loss

Network (notice the IP, it has the external IP - on compute node itll see it as the tenant IP!): IP!):

[root@network ~]#  tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo'
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
11:09:07.570023 IP (tos 0x0, ttl 63, id 50968, offset 0, flags [DF], proto ICMP (1), length 84)
    10.7.5.102 > 10.7.5.21: ICMP echo request, id 20481, seq 0, length 64
11:09:07.570032 ethertype IPv4, IP (tos 0x0, ttl 63, id 50968, offset 0, flags [DF], proto ICMP (1), length 84)
    10.7.5.102 > 10.7.5.21: ICMP echo request, id 20481, seq 0, length 64
11:09:07.570021 IP (tos 0x0, ttl 63, id 50968, offset 0, flags [DF], proto ICMP (1), length 84)
    10.7.5.102 > 10.7.5.21: ICMP echo request, id 20481, seq 0, length 64

64

From instance to compute node (notice the tenant IP):

Instance:

# ping 10.7.5.31

10.7.5.31 PING 10.7.5.31 (10.7.5.31): 56 data bytes 3 packets transmitted, 0 packets recieved, 100% packet lossloss

Compute1:

Compute1:

[root@compute1 ~]# tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo'
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

bytes 11:13:04.765748 IP (tos 0x0, ttl 64, id 43692, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.1.2 > 10.7.5.31: ICMP echo request, id 21249, seq 0, length 64 11:13:04.765780 IP (tos 0x0, ttl 64, id 43692, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.1.2 > 10.7.5.31: ICMP echo request, id 21249, seq 0, length 64 11:13:04.765782 IP (tos 0x0, ttl 64, id 43692, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.1.2 > 10.7.5.31: ICMP echo request, id 21249, seq 0, length 64

64

Pings to the controller node from the instances are not seen.