Revision history [back]

click to hide/show revision 1
initial version

Allocate IPv6 /64 block to each instance

The goal of this deployment is to allocate a /64 block of IPv6 addresses for each instance from a /48 block. To accomplish this we've been creating separate networks for each instance and adding a /64 subnet allocated through a subnet pool. The subnet is then routed through a gateway to the Public (provider) network.

IPv6 /48 addresses allocated directly through the Public network are working correctly, as are IPv4 addresses. However, The IPv6 /64 addresses are able to hit their router gateway, but cannot reach the outside network.

Ideally, we wanted to use a prefix delegation server such as dibbler, but we haven't been able to get the router advertisements to broadcast on the provider network. We fell back onto subnet pools in order to manage our IPv6 address allocation.

Public network - /48 block
- Router Gateway
  - Instance Network with a /64 block

Relevent Configs:

neutron.ini:

 [default]
 core_plugin = ml2
 service_plugins = router
 pd_interface = br-provider
 pd_dhcp_driver = neutron_pd_agent
 default_ipv6_subnet_pool = prefix_delegation

l3_agent.ini:

 [default]
 agent_mode = dvr
 interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
 ipv6_gateway = fe80::212:f2ff:fe91:b800 # Upstream routers link-local address

plugins/ml2/ml2_conf.ini

 [default]
 type_drivers = gre,flat
 tenant_network_types = gre,flat
 mechanism_drivers = openvswitch
 flat_networks = provider

 [securitygroup]
 bridge_mappings = provider:br-provider
 tunnel_type = gre
 enable_tunneling = True

 [agent]
 l2_population = True
 tunnel_types = gre
 enable_distributed_routing = True
 arp_responder = True
 prevent_arp_spoofing = True

/etc/dibbler/server.conf

iface "br-provider" {
 pd-class {
     pd-pool 2001:db8:9::/48
     pd-length 64
 }
}

Create the Public Network (under the Admin Project, 45a535ca3cb74cdca857ee1ae4efa0fe): neutron net-create Public --provider:network_type flat --provider:physical_network provider --router:external --share

+---------------------------+----------------------------------------------------------------------------+
| Field                     | Value                                                                      |
+---------------------------+----------------------------------------------------------------------------+
| admin_state_up            | UP                                                                         |
| availability_zone_hints   |                                                                            |
| availability_zones        | nova                                                                       |
| created_at                | 2017-12-19T09:23:42Z                                                       |
| description               |                                                                            |
| dns_domain                | None                                                                       |
| id                        | 3ea1c1c7-4eaf-4d64-a45e-d16e6d326934                                       |
| ipv4_address_scope        | None                                                                       |
| ipv6_address_scope        | None                                                                       |
| is_default                | False                                                                      |
| mtu                       | 1500                                                                       |
| name                      | Public                                                                     |
| port_security_enabled     | False                                                                      |
| project_id                | 45a535ca3cb74cdca857ee1ae4efa0fe                                           |
| provider:network_type     | flat                                                                       |
| provider:physical_network | provider                                                                   |
| provider:segmentation_id  | None                                                                       |
| qos_policy_id             | None                                                                       |
| revision_number           | 7                                                                          |
| router:external           | External                                                                   |
| segments                  | None                                                                       |
| shared                    | True                                                                       |
| status                    | ACTIVE                                                                     |
| subnets                   | 4778bf72-9777-434f-b5e7-2f358aba155e, e1658d2f-95d5-45b0-818e-6ac63aeba619 |
| updated_at                | 2017-12-19T09:24:35Z                                                       |
+---------------------------+----------------------------------------------------------------------------+

Add IPv4/IPv6 Subnets to the public network neutron subnet-create Public --ip-version 6 2001:db8:9::/48 --allocation-pool start=2001:0db8:0009:0010:0000:0000:0000:0000,end=2001:0db8:0009:ffff:ffff:ffff:ffff:fffe

+-------------------+-----------------------------------------------------+
| Field             | Value                                               |
+-------------------+-----------------------------------------------------+
| allocation_pools  | 2001:db8:9:10::-2001:db8:9:ffff:ffff:ffff:ffff:fffe |
| cidr              | 2001:db8:9::/48                                     |
| created_at        | 2017-12-19T09:24:25Z                                |
| description       |                                                     |
| dns_nameservers   |                                                     |
| enable_dhcp       | True                                                |
| gateway_ip        | 2001:db8:9::1                                       |
| host_routes       |                                                     |
| id                | e1658d2f-95d5-45b0-818e-6ac63aeba619                |
| ip_version        | 6                                                   |
| ipv6_address_mode | None                                                |
| ipv6_ra_mode      | None                                                |
| name              |                                                     |
| network_id        | 3ea1c1c7-4eaf-4d64-a45e-d16e6d326934                |
| project_id        | 45a535ca3cb74cdca857ee1ae4efa0fe                    |
| revision_number   | 2                                                   |
| segment_id        | None                                                |
| service_types     |                                                     |
| subnetpool_id     | None                                                |
| updated_at        | 2017-12-19T09:24:25Z                                |
+-------------------+-----------------------------------------------------+

neutron subnet-create Public --ip-version 4 192.0.2.128/25 --allocation-pool start=192.0.2.130,end=192.0.2.254 --enable-dhcp --dns_nameservers list=true 8.8.4.4 8.8.8.8

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 192.0.2.130-192.0.2.254      |
| cidr              | 192.0.2.128/25                   |
| created_at        | 2017-12-19T09:24:35Z                 |
| description       |                                      |
| dns_nameservers   | 8.8.4.4, 8.8.8.8                     |
| enable_dhcp       | True                                 |
| gateway_ip        | 192.0.2.129                      |
| host_routes       |                                      |
| id                | 4778bf72-9777-434f-b5e7-2f358aba155e |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              |                                      |
| network_id        | 3ea1c1c7-4eaf-4d64-a45e-d16e6d326934 |
| project_id        | 45a535ca3cb74cdca857ee1ae4efa0fe     |
| revision_number   | 2                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| updated_at        | 2017-12-19T09:24:35Z                 |
+-------------------+--------------------------------------+

Create IPv6 Subnet Pool. openstack subnet pool create --pool-prefix 2001:db8:9::/48 --share --prefix 64 public-ipv6-pool --default

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| address_scope_id  | None                                 |
| created_at        | 2017-12-12T19:40:36Z                 |
| default_prefixlen | 64                                   |
| default_quota     | None                                 |
| description       |                                      |
| id                | 8d238028-02e7-4d07-b6af-e3b94af8dc5b |
| ip_version        | 6                                    |
| is_default        | True                                 |
| max_prefixlen     | 128                                  |
| min_prefixlen     | 64                                   |
| name              | public-ipv6-pool                     |
| prefixes          | 2001:db8:9::/48                      |
| project_id        | 45a535ca3cb74cdca857ee1ae4efa0fe     |
| revision_number   | 1                                    |
| shared            | True                                 |
| updated_at        | 2017-12-12T19:40:36Z                 |
+-------------------+--------------------------------------+

Create Instance IPv6 Router neutron router-create dist-router-test-1 --distributed neutron router-gateway-set dist-router-test-1 Public

+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| admin_state_up          | True                                 |
| availability_zone_hints |                                      |
| availability_zones      |                                      |
| created_at              | 2017-12-15T22:27:59Z                 |
| description             |                                      |
| distributed             | False                                |
| external_gateway_info   |                                      |
| flavor_id               |                                      |
| ha                      | False                                |
| id                      | 929a3f76-be15-44e9-85e2-8bf949655825 |
| name                    | dist-router-test-1                   |
| project_id              | 45a535ca3cb74cdca857ee1ae4efa0fe     |
| revision_number         | 3                                    |
| routes                  |                                      |
| status                  | ACTIVE                               |
| tags                    |                                      |
| tenant_id               | 45a535ca3cb74cdca857ee1ae4efa0fe     |
| updated_at              | 2017-12-15T22:27:59Z                 |
+-------------------------+--------------------------------------+

Create Private Instance Network neutron net-create instance-network-test-1

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2017-12-15T22:28:10Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 5c21c24f-2090-4a3d-ac80-994be2292734 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| mtu                       | 1458                                 |
| name                      | instance-network-test-1              |
| port_security_enabled     | False                                |
| project_id                | 45a535ca3cb74cdca857ee1ae4efa0fe     |
| provider:network_type     | gre                                  |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 203                                  |
| qos_policy_id             | None                                 |
| revision_number           | 2                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| updated_at                | 2017-12-15T22:28:10Z                 |
+---------------------------+--------------------------------------+

Add IPv6 Subnet to Private Instance Network neutron subnet-create instance-network-test-1 --name instance-ipv6-subnet-1 --ip_version 6 --ipv6_ra_mode slaac --ipv6_address_mode slaac --use-default-subnetpool

+-------------------+-------------------------------------------------------------------------+
| Field             | Value                                                                   |
+-------------------+-------------------------------------------------------------------------+
| allocation_pools  | {"start": "2001:db8:9:2::2", "end": "2001:db8:9:2:ffff:ffff:ffff:ffff"} |
| cidr              | 2001:db8:9:2::/64                                                       |
| created_at        | 2017-12-19T09:29:13Z                                                    |
| description       |                                                                         |
| dns_nameservers   |                                                                         |
| enable_dhcp       | True                                                                    |
| gateway_ip        | 2001:db8:9:2::1                                                         |
| host_routes       |                                                                         |
| id                | ab841e80-9e45-4f0d-8b7a-410094d69857                                    |
| ip_version        | 6                                                                       |
| ipv6_address_mode | slaac                                                                   |
| ipv6_ra_mode      | slaac                                                                   |
| name              | instance-ipv6-subnet-1                                                  |
| network_id        | 5c21c24f-2090-4a3d-ac80-994be2292734                                    |
| project_id        | 45a535ca3cb74cdca857ee1ae4efa0fe                                        |
| revision_number   | 2                                                                       |
| service_types     |                                                                         |
| subnetpool_id     | 8d238028-02e7-4d07-b6af-e3b94af8dc5b                                    |
| tags              |                                                                         |
| tenant_id         | 45a535ca3cb74cdca857ee1ae4efa0fe                                        |
| updated_at        | 2017-12-19T09:29:13Z                                                    |
+-------------------+-------------------------------------------------------------------------+

Route IPv6 Subnet through IPv6 Router openstack router add subnet dist-router-test-1

Network Layout

enp4s0f0 - Management Network
enp4s0f1 - Instance Network (bridged to br-provider)
br-int (peer bridge to br-provider)

Compute Node ifconfig:

enp4s0f0  Link encap:Ethernet  HWaddr 0c:c4:7a:28:83:b0
          inet addr:192.0.2.3  Bcast:192.0.2.31  Mask:255.255.255.224
          inet6 addr: fe80::ec4:7aff:fe28:83b0/64 Scope:Link
          inet6 addr: 2001:db8:c:2::1/48 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4165513 errors:0 dropped:0 overruns:0 frame:0
          TX packets:187722 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:327024573 (327.0 MB)  TX bytes:77246505 (77.2 MB)
          Memory:c7420000-c743ffff

enp4s0f1  Link encap:Ethernet  HWaddr 0c:c4:7a:28:83:b1
          inet6 addr: fe80::ec4:7aff:fe28:83b1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1055589 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8648 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:139377082 (139.3 MB)  TX bytes:1089063 (1.0 MB)
          Memory:c7400000-c741ffff

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1587278 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1587278 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:116359494 (116.3 MB)  TX bytes:116359494 (116.3 MB)

qbr603cb9c9-a4 Link encap:Ethernet  HWaddr 8a:57:7c:02:3a:e5
          UP BROADCAST RUNNING MULTICAST  MTU:1458  Metric:1
          RX packets:949 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:87340 (87.3 KB)  TX bytes:0 (0.0 B)

qbre9f4072f-ce Link encap:Ethernet  HWaddr c2:92:e2:74:fa:75
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:922220 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:42878574 (42.8 MB)  TX bytes:0 (0.0 B)

qvb603cb9c9-a4 Link encap:Ethernet  HWaddr 8a:57:7c:02:3a:e5
          inet6 addr: fe80::8857:7cff:fe02:3ae5/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1458  Metric:1
          RX packets:1078 errors:0 dropped:0 overruns:0 frame:0
          TX packets:152 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:112196 (112.1 KB)  TX bytes:13836 (13.8 KB)

qvbe9f4072f-ce Link encap:Ethernet  HWaddr c2:92:e2:74:fa:75
          inet6 addr: fe80::c092:e2ff:fe74:fa75/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:938341 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7263 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:131562703 (131.5 MB)  TX bytes:859209 (859.2 KB)

qvo603cb9c9-a4 Link encap:Ethernet  HWaddr 96:f5:23:30:35:7f
          inet6 addr: fe80::94f5:23ff:fe30:357f/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1458  Metric:1
          RX packets:152 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1078 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13836 (13.8 KB)  TX bytes:112196 (112.1 KB)

qvoe9f4072f-ce Link encap:Ethernet  HWaddr 06:8e:30:d7:f2:b4
          inet6 addr: fe80::48e:30ff:fed7:f2b4/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:7263 errors:0 dropped:0 overruns:0 frame:0
          TX packets:938341 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:859209 (859.2 KB)  TX bytes:131562703 (131.5 MB)

tap603cb9c9-a4 Link encap:Ethernet  HWaddr fe:16:3e:c0:48:3f
          inet6 addr: fe80::fc16:3eff:fec0:483f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1458  Metric:1
          RX packets:145 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1082 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13098 (13.0 KB)  TX bytes:113396 (113.3 KB)

tape9f4072f-ce Link encap:Ethernet  HWaddr fe:16:3e:24:0c:75
          inet6 addr: fe80::fc16:3eff:fe24:c75/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7256 errors:0 dropped:0 overruns:0 frame:0
          TX packets:937298 errors:0 dropped:1046 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:858471 (858.4 KB)  TX bytes:131488225 (131.4 MB)

virbr0    Link encap:Ethernet  HWaddr 52:54:00:41:44:e5
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

8c889600-bf2a-408a-8242-39a7555793ae.radvd.conf

interface qr-eb4f58e1-89
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;


   AdvLinkMTU 1458;
   prefix 2001:db8:9:2::/64
   {
        AdvOnLink on;
        AdvAutonomous on;
   };
};

sudo ovs-appctl dpif/show system@ovs-system: hit:727170 missed:326722

br-ex:
        br-ex 65534/4: (internal)
br-int:
        br-int 65534/3: (internal)
        int-br-provider 1/none: (patch: peer=phy-br-provider)
        patch-tun 2/none: (patch: peer=patch-int)
        qr-eb4f58e1-89 19/8: (internal)
        qvo603cb9c9-a4 18/2: (system)
        qvoe9f4072f-ce 17/1: (system)
br-provider:
        br-provider 65534/7: (internal)
        enp4s0f1 1/6: (system)
        phy-br-provider 2/none: (patch: peer=int-br-provider)
br-tun:
        br-tun 65534/5: (internal)
        patch-int 1/none: (patch: peer=patch-tun)

sudo ovs-vsctl show

aa5ef60c-e47a-445d-bad7-ec2e4112b61b
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qvoe9f4072f-ce"
            tag: 5
            Interface "qvoe9f4072f-ce"
        Port "qr-eb4f58e1-89"
            tag: 6
            Interface "qr-eb4f58e1-89"
                type: internal
        Port "qvo603cb9c9-a4"
            tag: 6
            Interface "qvo603cb9c9-a4"
        Port br-int
            Interface br-int
                type: internal
        Port int-br-provider
            Interface int-br-provider
                type: patch
                options: {peer=phy-br-provider}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-provider
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-provider
            Interface br-provider
                type: internal
        Port phy-br-provider
            Interface phy-br-provider
                type: patch
                options: {peer=int-br-provider}
    ovs_version: "2.6.1"

sudo ovs-appctl ovs/route/show Route Table:

Cached: 127.0.0.1/32 dev lo SRC 127.0.0.1
Cached: 192.0.2.3/32 dev enp4s0f0 SRC 192.0.2.3
Cached: 192.168.122.1/32 dev virbr0 SRC 192.168.122.1
Cached: ::1/128 dev lo SRC ::1
Cached: 2001:db8:c::/128 dev lo SRC ::1
Cached: 2001:db8:c:2::1/128 dev lo SRC ::1
Cached: fe80::/128 dev lo SRC ::1
Cached: fe80::48e:30ff:fed7:f2b4/128 dev lo SRC ::1
Cached: fe80::ec4:7aff:fe28:83b0/128 dev lo SRC ::1
Cached: fe80::ec4:7aff:fe28:83b1/128 dev lo SRC ::1
Cached: fe80::8857:7cff:fe02:3ae5/128 dev lo SRC ::1
Cached: fe80::94f5:23ff:fe30:357f/128 dev lo SRC ::1
Cached: fe80::c092:e2ff:fe74:fa75/128 dev lo SRC ::1
Cached: fe80::fc16:3eff:fe24:c75/128 dev lo SRC ::1
Cached: fe80::fc16:3eff:fec0:483f/128 dev lo SRC ::1
Cached: 192.0.2.0/27 dev enp4s0f0 SRC 192.0.2.3
Cached: 192.168.122.0/24 dev virbr0 SRC 192.168.122.1
Cached: 127.0.0.0/8 dev lo SRC 127.0.0.1
Cached: 0.0.0.0/0 dev enp4s0f0 GW 192.0.2.1 SRC 192.0.2.3
Cached: fe80::/64 dev enp4s0f1 SRC fe80::ec4:7aff:fe28:83b1
Cached: 2001:db8:c::/48 dev enp4s0f0 SRC 2001:db8:c:2::1
Cached: ff00::/8 dev enp4s0f1 SRC fe80::ec4:7aff:fe28:83b1
Cached: ::/0 dev enp4s0f0 GW 2001:db8:c::1 SRC 2001:db8:c:2::1