Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Liberty RDO "Neutron with existing external network"

I've been trying to do "Neutron with existing external network" install by using the latest centos 7 + RDO. When booting a Fedora 23 instance, I'm stuck with this (cirros has similar symptoms as well):

ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
ci-info: |  lo:   | True |     .     |     .     |   d   |         .         |
ci-info: | eth0:  | True |     .     |     .     |   .   | fa:16:3e:34:71:a3 |
ci-info: | eth0:  | True |     .     |     .     |   d   | fa:16:3e:34:71:a3 |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', OSError(101, 'Network is unreachable'))]

What I did was:

  1. Follow instructions in https://www.rdoproject.org/networking/neutron-with-existing-external-network/ (except added "--os-horizon-ssl=y --ntp-servers=..." to packstack and modified /etc/neutron/plugins/ml2/openvswitch_agent.ini and /etc/neutron/plugins/ml2/ml2_conf.ini as instructed in https://ask.openstack.org/en/question/86601/neutron-with-existing-external-network/ (except replaced physnet1 by extnet)).

  2. Fixed the quota problem: https://ask.openstack.org/en/question/87002/cinder-volume-quotas-broken-for-rdo-install-in-centos/

  3. Tried to create a Fedora/Cirros instance: https://www.rdoproject.org/networking/running-an-instance-with-neutron/

I can ping the first address of the floating IP range, but not the assigned one.

# ifconfig
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.50  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 2001:db8::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x20<link>
        ether 78:2b:cb:47:f0:37  txqueuelen 0  (Ethernet)
        RX packets 17433  bytes 2293095 (2.1 MiB)
        RX errors 0  dropped 15  overruns 0  frame 0
        TX packets 2751  bytes 2667610 (2.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x20<link>
        ether 78:2b:cb:47:f0:37  txqueuelen 1000  (Ethernet)
        RX packets 18641  bytes 2452312 (2.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3904  bytes 2760240 (2.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 286112  bytes 44247917 (42.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 286112  bytes 44247917 (42.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qbrd0bd198a-f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::643b:4ff:fe60:bbfe  prefixlen 64  scopeid 0x20<link>
        ether 6a:cc:4a:ac:f1:f7  txqueuelen 0  (Ethernet)
        RX packets 28  bytes 5260 (5.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvbd0bd198a-f7: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::68cc:4aff:feac:f1f7  prefixlen 64  scopeid 0x20<link>
        ether 6a:cc:4a:ac:f1:f7  txqueuelen 1000  (Ethernet)
        RX packets 8  bytes 648 (648.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36  bytes 6300 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvod0bd198a-f7: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::e019:e8ff:fe31:eef2  prefixlen 64  scopeid 0x20<link>
        ether e2:19:e8:31:ee:f2  txqueuelen 1000  (Ethernet)
        RX packets 36  bytes 6300 (6.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tapd0bd198a-f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc16:3eff:fe34:71a3  prefixlen 64  scopeid 0x20<link>
        ether fe:16:3e:34:71:a3  txqueuelen 500  (Ethernet)
        RX packets 21  bytes 5094 (4.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18  bytes 1456 (1.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ovs-vsctl show
2d33fb90-fcf5-4a96-9b49-1c5907626566
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qvod0bd198a-f7"
            tag: 2
            Interface "qvod0bd198a-f7"
        Port "qr-6241f38b-7e"
            tag: 4095
            Interface "qr-6241f38b-7e"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "tapc6c30ac8-01"
            tag: 4095
            Interface "tapc6c30ac8-01"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-ex
        Port "em1"
            Interface "em1"
        Port "qg-5939f02e-f0"
            Interface "qg-5939f02e-f0"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.4.0"
# neutron net-list
+--------------------------------------+------------------+-------------------------------------------------------+
| id                                   | name             | subnets                                               |
+--------------------------------------+------------------+-------------------------------------------------------+
| effa1df5-b19f-4325-9701-f4be01cfd2a1 | external_network | 9c7110fb-fbdf-4926-a44b-0487491a0cd5 192.168.122.0/24 |
| 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 | private_network  | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688 192.168.100.0/24 |
+--------------------------------------+------------------+-------------------------------------------------------+
# neutron net-show effa1df5-b19f-4325-9701-f4be01cfd2a1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | effa1df5-b19f-4325-9701-f4be01cfd2a1 |
| mtu                       | 0                                    |
| name                      | external_network                     |
| provider:network_type     | flat                                 |
| provider:physical_network | extnet                               |
| provider:segmentation_id  |                                      |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 9c7110fb-fbdf-4926-a44b-0487491a0cd5 |
| tenant_id                 | 4b073dc699504460920b7f1a4dcf4653     |
+---------------------------+--------------------------------------+
# neutron subnet-show 9c7110fb-fbdf-4926-a44b-0487491a0cd5
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "192.168.122.57", "end": "192.168.122.59"} |
| cidr              | 192.168.122.0/24                                     |
| dns_nameservers   |                                                      |
| enable_dhcp       | False                                                |
| gateway_ip        | 192.168.122.1                                        |
| host_routes       |                                                      |
| id                | 9c7110fb-fbdf-4926-a44b-0487491a0cd5                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | public_subnet                                        |
| network_id        | effa1df5-b19f-4325-9701-f4be01cfd2a1                 |
| subnetpool_id     |                                                      |
| tenant_id         | 4b073dc699504460920b7f1a4dcf4653                     |
+-------------------+------------------------------------------------------+
# neutron net-show 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 |
| mtu                       | 0                                    |
| name                      | private_network                      |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 23                                   |
| router:external           | False                                |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688 |
| tenant_id                 | 4b073dc699504460920b7f1a4dcf4653     |
+---------------------------+--------------------------------------+
# neutron subnet-show 57fb6853-4562-4bcb-b3b3-fd4b85bc3688
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "192.168.100.2", "end": "192.168.100.254"} |
| cidr              | 192.168.100.0/24                                     |
| dns_nameservers   |                                                      |
| enable_dhcp       | True                                                 |
| gateway_ip        | 192.168.100.1                                        |
| host_routes       |                                                      |
| id                | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | private_subnet                                       |
| network_id        | 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8                 |
| subnetpool_id     |                                                      |
| tenant_id         | 4b073dc699504460920b7f1a4dcf4653                     |
+-------------------+------------------------------------------------------+

Liberty RDO "Neutron with existing external network"

I've been trying to do "Neutron with existing external network" install by using the latest centos 7 + RDO. When booting a Fedora 23 instance, I'm stuck with this (cirros has similar symptoms as well):

ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
ci-info: |  lo:   | True |     .     |     .     |   d   |         .         |
ci-info: | eth0:  | True |     .     |     .     |   .   | fa:16:3e:34:71:a3 |
ci-info: | eth0:  | True |     .     |     .     |   d   | fa:16:3e:34:71:a3 |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', OSError(101, 'Network is unreachable'))]

What I did was:

  1. Follow instructions in https://www.rdoproject.org/networking/neutron-with-existing-external-network/ (except added "--os-horizon-ssl=y --ntp-servers=..." to packstack and modified /etc/neutron/plugins/ml2/openvswitch_agent.ini and /etc/neutron/plugins/ml2/ml2_conf.ini as instructed in https://ask.openstack.org/en/question/86601/neutron-with-existing-external-network/ (except replaced physnet1 by extnet)).

  2. Fixed the quota problem: https://ask.openstack.org/en/question/87002/cinder-volume-quotas-broken-for-rdo-install-in-centos/

  3. Tried to create a Fedora/Cirros instance: https://www.rdoproject.org/networking/running-an-instance-with-neutron/

I can ping the first address of the floating IP range, but not the assigned one.

# ifconfig
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.50  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 2001:db8::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x20<link>
        ether 78:2b:cb:47:f0:37  txqueuelen 0  (Ethernet)
        RX packets 17433  bytes 2293095 (2.1 MiB)
        RX errors 0  dropped 15  overruns 0  frame 0
        TX packets 2751  bytes 2667610 (2.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x20<link>
        ether 78:2b:cb:47:f0:37  txqueuelen 1000  (Ethernet)
        RX packets 18641  bytes 2452312 (2.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3904  bytes 2760240 (2.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 286112  bytes 44247917 (42.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 286112  bytes 44247917 (42.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qbrd0bd198a-f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::643b:4ff:fe60:bbfe  prefixlen 64  scopeid 0x20<link>
        ether 6a:cc:4a:ac:f1:f7  txqueuelen 0  (Ethernet)
        RX packets 28  bytes 5260 (5.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvbd0bd198a-f7: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::68cc:4aff:feac:f1f7  prefixlen 64  scopeid 0x20<link>
        ether 6a:cc:4a:ac:f1:f7  txqueuelen 1000  (Ethernet)
        RX packets 8  bytes 648 (648.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36  bytes 6300 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvod0bd198a-f7: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::e019:e8ff:fe31:eef2  prefixlen 64  scopeid 0x20<link>
        ether e2:19:e8:31:ee:f2  txqueuelen 1000  (Ethernet)
        RX packets 36  bytes 6300 (6.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tapd0bd198a-f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc16:3eff:fe34:71a3  prefixlen 64  scopeid 0x20<link>
        ether fe:16:3e:34:71:a3  txqueuelen 500  (Ethernet)
        RX packets 21  bytes 5094 (4.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18  bytes 1456 (1.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ovs-vsctl show
2d33fb90-fcf5-4a96-9b49-1c5907626566
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qvod0bd198a-f7"
            tag: 2
            Interface "qvod0bd198a-f7"
        Port "qr-6241f38b-7e"
            tag: 4095
            Interface "qr-6241f38b-7e"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "tapc6c30ac8-01"
            tag: 4095
            Interface "tapc6c30ac8-01"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-ex
        Port "em1"
            Interface "em1"
        Port "qg-5939f02e-f0"
            Interface "qg-5939f02e-f0"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.4.0"
# neutron net-list
+--------------------------------------+------------------+-------------------------------------------------------+
| id                                   | name             | subnets                                               |
+--------------------------------------+------------------+-------------------------------------------------------+
| effa1df5-b19f-4325-9701-f4be01cfd2a1 | external_network | 9c7110fb-fbdf-4926-a44b-0487491a0cd5 192.168.122.0/24 |
| 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 | private_network  | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688 192.168.100.0/24 |
+--------------------------------------+------------------+-------------------------------------------------------+
# neutron net-show effa1df5-b19f-4325-9701-f4be01cfd2a1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | effa1df5-b19f-4325-9701-f4be01cfd2a1 |
| mtu                       | 0                                    |
| name                      | external_network                     |
| provider:network_type     | flat                                 |
| provider:physical_network | extnet                               |
| provider:segmentation_id  |                                      |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 9c7110fb-fbdf-4926-a44b-0487491a0cd5 |
| tenant_id                 | 4b073dc699504460920b7f1a4dcf4653     |
+---------------------------+--------------------------------------+
# neutron subnet-show 9c7110fb-fbdf-4926-a44b-0487491a0cd5
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "192.168.122.57", "end": "192.168.122.59"} |
| cidr              | 192.168.122.0/24                                     |
| dns_nameservers   |                                                      |
| enable_dhcp       | False                                                |
| gateway_ip        | 192.168.122.1                                        |
| host_routes       |                                                      |
| id                | 9c7110fb-fbdf-4926-a44b-0487491a0cd5                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | public_subnet                                        |
| network_id        | effa1df5-b19f-4325-9701-f4be01cfd2a1                 |
| subnetpool_id     |                                                      |
| tenant_id         | 4b073dc699504460920b7f1a4dcf4653                     |
+-------------------+------------------------------------------------------+
# neutron net-show 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 |
| mtu                       | 0                                    |
| name                      | private_network                      |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 23                                   |
| router:external           | False                                |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688 |
| tenant_id                 | 4b073dc699504460920b7f1a4dcf4653     |
+---------------------------+--------------------------------------+
# neutron subnet-show 57fb6853-4562-4bcb-b3b3-fd4b85bc3688
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "192.168.100.2", "end": "192.168.100.254"} |
| cidr              | 192.168.100.0/24                                     |
| dns_nameservers   |                                                      |
| enable_dhcp       | True                                                 |
| gateway_ip        | 192.168.100.1                                        |
| host_routes       |                                                      |
| id                | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | private_subnet                                       |
| network_id        | 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8                 |
| subnetpool_id     |                                                      |
| tenant_id         | 4b073dc699504460920b7f1a4dcf4653                     |
+-------------------+------------------------------------------------------+

EDIT: Can't access default router inside qdhcp namespace:

# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 ip r
default via 192.168.100.1 dev tapc6c30ac8-01
192.168.100.0/24 dev tapc6c30ac8-01  proto kernel  scope link  src 192.168.100.2
# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=0.052 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.058 ms
...
# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable  From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
...
# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 tcpdump -ni tapc6c30ac8-01
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapc6c30ac8-01, link-type EN10MB (Ethernet), capture size
65535 bytes
10:07:45.212281 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
10:07:46.214676 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
...

Liberty RDO "Neutron with existing external network"

I've been trying to do "Neutron with existing external network" install by using the latest centos 7 + RDO. When booting a Fedora 23 instance, I'm stuck with this (cirros has similar symptoms as well):

ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
ci-info: |  lo:   | True |     .     |     .     |   d   |         .         |
ci-info: | eth0:  | True |     .     |     .     |   .   | fa:16:3e:34:71:a3 |
ci-info: | eth0:  | True |     .     |     .     |   d   | fa:16:3e:34:71:a3 |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', OSError(101, 'Network is unreachable'))]

What I did was:

  1. Follow instructions in https://www.rdoproject.org/networking/neutron-with-existing-external-network/ (except added "--os-horizon-ssl=y --ntp-servers=..." to packstack and modified /etc/neutron/plugins/ml2/openvswitch_agent.ini and /etc/neutron/plugins/ml2/ml2_conf.ini as instructed in https://ask.openstack.org/en/question/86601/neutron-with-existing-external-network/ (except replaced physnet1 by extnet)).

  2. Fixed the quota problem: https://ask.openstack.org/en/question/87002/cinder-volume-quotas-broken-for-rdo-install-in-centos/

  3. Tried to create a Fedora/Cirros instance: https://www.rdoproject.org/networking/running-an-instance-with-neutron/

I can ping the first address of the floating IP range, but not the assigned one.

# ifconfig
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.50  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 2001:db8::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x20<link>
        ether 78:2b:cb:47:f0:37  txqueuelen 0  (Ethernet)
        RX packets 17433  bytes 2293095 (2.1 MiB)
        RX errors 0  dropped 15  overruns 0  frame 0
        TX packets 2751  bytes 2667610 (2.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::7a2b:cbff:fe47:f037  prefixlen 64  scopeid 0x20<link>
        ether 78:2b:cb:47:f0:37  txqueuelen 1000  (Ethernet)
        RX packets 18641  bytes 2452312 (2.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3904  bytes 2760240 (2.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 286112  bytes 44247917 (42.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 286112  bytes 44247917 (42.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qbrd0bd198a-f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::643b:4ff:fe60:bbfe  prefixlen 64  scopeid 0x20<link>
        ether 6a:cc:4a:ac:f1:f7  txqueuelen 0  (Ethernet)
        RX packets 28  bytes 5260 (5.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvbd0bd198a-f7: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::68cc:4aff:feac:f1f7  prefixlen 64  scopeid 0x20<link>
        ether 6a:cc:4a:ac:f1:f7  txqueuelen 1000  (Ethernet)
        RX packets 8  bytes 648 (648.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36  bytes 6300 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvod0bd198a-f7: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::e019:e8ff:fe31:eef2  prefixlen 64  scopeid 0x20<link>
        ether e2:19:e8:31:ee:f2  txqueuelen 1000  (Ethernet)
        RX packets 36  bytes 6300 (6.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tapd0bd198a-f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc16:3eff:fe34:71a3  prefixlen 64  scopeid 0x20<link>
        ether fe:16:3e:34:71:a3  txqueuelen 500  (Ethernet)
        RX packets 21  bytes 5094 (4.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18  bytes 1456 (1.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ovs-vsctl show
2d33fb90-fcf5-4a96-9b49-1c5907626566
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qvod0bd198a-f7"
            tag: 2
            Interface "qvod0bd198a-f7"
        Port "qr-6241f38b-7e"
            tag: 4095
            Interface "qr-6241f38b-7e"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "tapc6c30ac8-01"
            tag: 4095
            Interface "tapc6c30ac8-01"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-ex
        Port "em1"
            Interface "em1"
        Port "qg-5939f02e-f0"
            Interface "qg-5939f02e-f0"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.4.0"
# neutron net-list
+--------------------------------------+------------------+-------------------------------------------------------+
| id                                   | name             | subnets                                               |
+--------------------------------------+------------------+-------------------------------------------------------+
| effa1df5-b19f-4325-9701-f4be01cfd2a1 | external_network | 9c7110fb-fbdf-4926-a44b-0487491a0cd5 192.168.122.0/24 |
| 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 | private_network  | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688 192.168.100.0/24 |
+--------------------------------------+------------------+-------------------------------------------------------+
# neutron net-show effa1df5-b19f-4325-9701-f4be01cfd2a1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | effa1df5-b19f-4325-9701-f4be01cfd2a1 |
| mtu                       | 0                                    |
| name                      | external_network                     |
| provider:network_type     | flat                                 |
| provider:physical_network | extnet                               |
| provider:segmentation_id  |                                      |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 9c7110fb-fbdf-4926-a44b-0487491a0cd5 |
| tenant_id                 | 4b073dc699504460920b7f1a4dcf4653     |
+---------------------------+--------------------------------------+
# neutron subnet-show 9c7110fb-fbdf-4926-a44b-0487491a0cd5
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "192.168.122.57", "end": "192.168.122.59"} |
| cidr              | 192.168.122.0/24                                     |
| dns_nameservers   |                                                      |
| enable_dhcp       | False                                                |
| gateway_ip        | 192.168.122.1                                        |
| host_routes       |                                                      |
| id                | 9c7110fb-fbdf-4926-a44b-0487491a0cd5                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | public_subnet                                        |
| network_id        | effa1df5-b19f-4325-9701-f4be01cfd2a1                 |
| subnetpool_id     |                                                      |
| tenant_id         | 4b073dc699504460920b7f1a4dcf4653                     |
+-------------------+------------------------------------------------------+
# neutron net-show 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 |
| mtu                       | 0                                    |
| name                      | private_network                      |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 23                                   |
| router:external           | False                                |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688 |
| tenant_id                 | 4b073dc699504460920b7f1a4dcf4653     |
+---------------------------+--------------------------------------+
# neutron subnet-show 57fb6853-4562-4bcb-b3b3-fd4b85bc3688
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "192.168.100.2", "end": "192.168.100.254"} |
| cidr              | 192.168.100.0/24                                     |
| dns_nameservers   |                                                      |
| enable_dhcp       | True                                                 |
| gateway_ip        | 192.168.100.1                                        |
| host_routes       |                                                      |
| id                | 57fb6853-4562-4bcb-b3b3-fd4b85bc3688                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | private_subnet                                       |
| network_id        | 6da75bca-39c1-4b68-a9bd-6cbf1ad945a8                 |
| subnetpool_id     |                                                      |
| tenant_id         | 4b073dc699504460920b7f1a4dcf4653                     |
+-------------------+------------------------------------------------------+

EDIT: Can't access default router inside qdhcp namespace:

# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 ip r
default via 192.168.100.1 dev tapc6c30ac8-01
192.168.100.0/24 dev tapc6c30ac8-01  proto kernel  scope link  src 192.168.100.2
# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=0.052 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.058 ms
...
# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable  From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
...
# ip netns exec qdhcp-6da75bca-39c1-4b68-a9bd-6cbf1ad945a8 tcpdump -ni tapc6c30ac8-01
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapc6c30ac8-01, link-type EN10MB (Ethernet), capture size
65535 bytes
10:07:45.212281 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
10:07:46.214676 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
...

EDIT 2:

# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=416229.745s, table=0, n_packets=69, n_bytes=7030, idle_age=65534, hard_age=65534, priority=2,in_port=2 actions=drop
  cookie=0x0, duration=416229.807s, table=0, n_packets=2710488, n_bytes=435694673, idle_age=0, hard_age=65534, priority=0 actions=NORMAL

# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
  cookie=0x9fc9b7ece24165a4, duration=6664.448s, table=0, n_packets=0, n_bytes=0, idle_age=6664, priority=10,icmp6,in_port=7,icmp_type=136 actions=resubmit(,24)
  cookie=0x9fc9b7ece24165a4, duration=5806.623s, table=0, n_packets=0, n_bytes=0, idle_age=5806, priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24)
  cookie=0x9fc9b7ece24165a4, duration=6664.434s, table=0, n_packets=0, n_bytes=0, idle_age=6664, priority=10,arp,in_port=7 actions=resubmit(,24)
  cookie=0x9fc9b7ece24165a4, duration=5806.610s, table=0, n_packets=0, n_bytes=0, idle_age=5806, priority=10,arp,in_port=8 actions=resubmit(,24)
  cookie=0x9fc9b7ece24165a4, duration=84766.376s, table=0, n_packets=455506, n_bytes=57678206, idle_age=0, hard_age=65534, priority=2,in_port=1 actions=drop
  cookie=0x9fc9b7ece24165a4, duration=84756.703s, table=0, n_packets=6, n_bytes=524, idle_age=65534, hard_age=65534, priority=2,in_port=4 actions=drop
  cookie=0x9fc9b7ece24165a4, duration=84704.776s, table=0, n_packets=68, n_bytes=3036, idle_age=6923, hard_age=65534, priority=2,in_port=6 actions=drop
  cookie=0x9fc9b7ece24165a4, duration=84766.528s, table=0, n_packets=70, n_bytes=7122, idle_age=5681, hard_age=65534, priority=0 actions=NORMAL
  cookie=0x9fc9b7ece24165a4, duration=84766.508s, table=23, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
  cookie=0x9fc9b7ece24165a4, duration=6664.454s, table=24, n_packets=0, n_bytes=0, idle_age=6664, priority=2,icmp6,in_port=7,icmp_type=136,nd_target=fe80::f816:3eff:fe16:efd7 actions=NORMAL
  cookie=0x9fc9b7ece24165a4, duration=5806.629s, table=24, n_packets=0, n_bytes=0, idle_age=5806, priority=2,icmp6,in_port=8,icmp_type=136,nd_target=fe80::f816:3eff:feee:46b6 actions=NORMAL
  cookie=0x9fc9b7ece24165a4, duration=6664.441s, table=24, n_packets=0, n_bytes=0, idle_age=6664, priority=2,arp,in_port=7,arp_spa=192.168.100.12 actions=NORMAL
  cookie=0x9fc9b7ece24165a4, duration=5806.617s, table=24, n_packets=0, n_bytes=0, idle_age=5806, priority=2,arp,in_port=8,arp_spa=192.168.100.13 actions=NORMAL
  cookie=0x9fc9b7ece24165a4, duration=84766.501s, table=24, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop