Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Floating IP not working (in-/egress from/to outside) [Neutron, OVS, GRE]

Trying to get Havana (rc1 from RDO) to work with Neutron using OVS and GRE tunnels. There's a single controller which also acts as the network node and there's currently two compute nodes. Trying to launch an instance, it gets a fixed IP which seems to work well. But then I assign a floating IP which is supposedly routed within our network. Now I can neither access our network from within the instance nor the other way around. I can't even ping the external gateway from within the instance and I can't ping the instance from the controller (and yes, I opened up the security group for all ICMP from 0.0.0.0/0).

Controller (aka ctrl below): - eth0 added to br-ex - br-ex has IP of 129.132.32.7 (a range that is not used anywhere else in the setup) - eth1 has the IP 10.112.1.30, which is the management network (over which the GRE tunnels go) - eth2 connects to our storage network and can be ignored in this scenario - Can ping Floating Gateway (external gateway) in default namespace - Can ping Floating Router and Floating IP in qrouter namespace - Can NOT ping Floating Gateway in qrouter namespace - Can NOT ping Floating Router or Floating IP in default namespace

Compute Nodes: - eth0 has IP of 10.112.1.{5,13}, which again is the management network (over which the GRE tunnels go)

Instance (aka instance below): - Private IP of 10.1.0.2 - Floating IP of 172.31.116.12 - Can ping itself on Private IP - Can ping Private Gateway - Can ping itself on Floating IP - Can ping Floating Router - Can NOT ping Floating Gateway

So what seems to sort of lack is a connection between the two namespaces (default and qrouter). But a tcpdump on eth0 reveals that if the instance tries to ping the Floating Gateway, there'a a arp request but there doesn't seem to be an answer. My guess would be because the network doesn't know where to send to answer to (i.e. the Floating IP is unknown to the network). Really no idea what I'm missing here.

Suspiciously, ALL ports on the Floating Network are shown as DOWN (while those on the Private Network are UP). I know there used to be a bug where they were shown down but actually worked, but this should be fixed and I think with the bug all ports on all networks showed as DOWN so not seeing that bug here.

Below, I'll list everything that I think could matter. Logs look clean, so leaving them away but happy to provide any additional information. Any help is much appreciated, thanks!

[root@ctrl]# neutron net-show eth_zurich
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 90696787-0c11-4031-bc99-8d1db2d24744 |
| name                      | eth_zurich                           |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 4                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 47e26025-f09a-4299-915e-6b2275fea59e |
| tenant_id                 | b6f22facd6bb43a5bb465097b53a087a     |
+---------------------------+--------------------------------------+

[root@ctrl]# neutron subnet-show private_v4
+------------------+---------------------------------------------------+
| Field            | Value                                             |
+------------------+---------------------------------------------------+
| allocation_pools | {"start": "172.31.116.7", "end": "172.31.116.59"} |
| cidr             | 172.31.116.0/24                                   |
| dns_nameservers  |                                                   |
| enable_dhcp      | False                                             |
| gateway_ip       | 172.31.116.1                                      |
| host_routes      |                                                   |
| id               | 47e26025-f09a-4299-915e-6b2275fea59e              |
| ip_version       | 4                                                 |
| name             | private_v4                                        |
| network_id       | 90696787-0c11-4031-bc99-8d1db2d24744              |
| tenant_id        | 76896fef704649c19d4fc921e9e2e680                  |
+------------------+---------------------------------------------------+

[root@ctrl]# neutron router-show portal
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "90696787-0c11-4031-bc99-8d1db2d24744", "enable_snat": true} |
| id                    | 0ca86e83-80fe-403e-83e2-d4652985cee6                                        |
| name                  | portal                                                                      |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | 76896fef704649c19d4fc921e9e2e680                                            |
+-----------------------+-----------------------------------------------------------------------------+

[root@ctrl]# neutron port-list
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                            |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 0d9dbb61-f8bf-4741-b94d-fcd835245747 |      | fa:16:3e:9c:4a:d8 | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.3"}      |
| 5366ad09-03ce-4426-8664-57cba7601833 |      | fa:16:3e:06:00:6c | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.7"}  |
| b9587621-f1d3-4268-ba5b-3a77c6f94037 |      | fa:16:3e:cb:47:8b | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.11"} |
| ca4cb00d-22b5-4973-b36f-63d0426cdc52 |      | fa:16:3e:0a:f5:c8 | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.1"}      |
| cf2eaeb4-5933-4d02-a8c1-969420e12632 |      | fa:16:3e:59:40:53 | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.12"} |
| d447ef74-70d2-4f13-a584-a6ac0c9d23d8 |      | fa:16:3e:68:6a:89 | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.2"}      |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+

[root@ctrl]# neutron port-show d447ef74-70d2-4f13-a584-a6ac0c9d23d8
+-----------------------+---------------------------------------------------------------------------------+
| Field                 | Value                                                                           |
+-----------------------+---------------------------------------------------------------------------------+
| admin_state_up        | True                                                                            |
| allowed_address_pairs |                                                                                 |
| binding:capabilities  | {"port_filter": true}                                                           |
| binding:host_id       | node05.ethz.lan                                                                 |
| binding:vif_type      | ovs                                                                             |
| device_id             | fd10e420-bab2-4209-95b7-7c18918ef48e                                            |
| device_owner          | compute:nova                                                                    |
| extra_dhcp_opts       |                                                                                 |
| fixed_ips             | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.2"} |
| id                    | d447ef74-70d2-4f13-a584-a6ac0c9d23d8                                            |
| mac_address           | fa:16:3e:68:6a:89                                                               |
| name                  |                                                                                 |
| network_id            | fa943fa6-6e44-46a8-a391-7f12a8249bf1                                            |
| security_groups       | 35953a34-2ebc-4f12-a3c7-3aeac6de2dc2                                            |
| status                | ACTIVE                                                                          |
| tenant_id             | 76896fef704649c19d4fc921e9e2e680                                                |
+-----------------------+---------------------------------------------------------------------------------+

[root@ctrl]# neutron port-show cf2eaeb4-5933-4d02-a8c1-969420e12632
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value                                                                                |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                 |
| allowed_address_pairs |                                                                                      |
| binding:capabilities  | {"port_filter": true}                                                                |
| binding:host_id       |                                                                                      |
| binding:vif_type      | ovs                                                                                  |
| device_id             | ae505a31-584d-465c-b922-69da4bc5f059                                                 |
| device_owner          | network:floatingip                                                                   |
| extra_dhcp_opts       |                                                                                      |
| fixed_ips             | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.12"} |
| id                    | cf2eaeb4-5933-4d02-a8c1-969420e12632                                                 |
| mac_address           | fa:16:3e:59:40:53                                                                    |
| name                  |                                                                                      |
| network_id            | 90696787-0c11-4031-bc99-8d1db2d24744                                                 |
| security_groups       |                                                                                      |
| status                | DOWN                                                                                 |
| tenant_id             |                                                                                      |
+-----------------------+--------------------------------------------------------------------------------------+

[root@ctrl]# neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 46731d68-3843-4642-b093-06816ba0246c |                  | 172.31.116.11       |                                      |
| ae505a31-584d-465c-b922-69da4bc5f059 | 10.1.0.2         | 172.31.116.12       | d447ef74-70d2-4f13-a584-a6ac0c9d23d8 |
+--------------------------------------+------------------+---------------------+--------------------------------------+

[root@ctrl]# neutron floatingip-show ae505a31-584d-465c-b922-69da4bc5f059
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.1.0.2                             |
| floating_ip_address | 172.31.116.12                        |
| floating_network_id | 90696787-0c11-4031-bc99-8d1db2d24744 |
| id                  | ae505a31-584d-465c-b922-69da4bc5f059 |
| port_id             | d447ef74-70d2-4f13-a584-a6ac0c9d23d8 |
| router_id           | 0ca86e83-80fe-403e-83e2-d4652985cee6 |
| tenant_id           | 76896fef704649c19d4fc921e9e2e680     |
+---------------------+--------------------------------------+

[root@ctrl]# nova list
+--------------------------------------+------+--------+------------+-------------+-----------------------------------+
| ID                                   | Name | Status | Task State | Power State | Networks                          |
+--------------------------------------+------+--------+------------+-------------+-----------------------------------+
| fd10e420-bab2-4209-95b7-7c18918ef48e | CT   | ACTIVE | None       | Running     | instances=10.1.0.2, 172.31.116.12 |
+--------------------------------------+------+--------+------------+-------------+-----------------------------------+

[root@ctrl]# nova show CT
+--------------------------------------+------------------------------------------------------------+
| Property                             | Value                                                      |
+--------------------------------------+------------------------------------------------------------+
| status                               | ACTIVE                                                     |
| updated                              | 2013-10-15T13:48:37Z                                       |
| OS-EXT-STS:task_state                | None                                                       |
| OS-EXT-SRV-ATTR:host                 | node05.ethz.lan                                            |
| key_name                             | None                                                       |
| image                                | Cirros 0.3.1 x86_64 (cfdfdaba-6d71-4be5-a17c-4b67ed18b130) |
| hostId                               | 2147b671c0f53d0bccbf737b305ac02ce48fe3aee9e4a2223df59fdd   |
| OS-EXT-STS:vm_state                  | active                                                     |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000000b                                          |
| OS-SRV-USG:launched_at               | 2013-10-15T13:48:37.000000                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node05.ethz.lan                                            |
| flavor                               | m1.tiny (1)                                                |
| id                                   | fd10e420-bab2-4209-95b7-7c18918ef48e                       |
| security_groups                      | [{u'name': u'default'}]                                    |
| OS-SRV-USG:terminated_at             | None                                                       |
| instances network                    | 10.1.0.2, 172.31.116.12                                    |
| user_id                              | e18119c0273d43b1b695094b746d3814                           |
| name                                 | CT                                                         |
| created                              | 2013-10-15T13:48:16Z                                       |
| tenant_id                            | 76896fef704649c19d4fc921e9e2e680                           |
| OS-DCF:diskConfig                    | MANUAL                                                     |
| metadata                             | {}                                                         |
| os-extended-volumes:volumes_attached | []                                                         |
| accessIPv4                           |                                                            |
| accessIPv6                           |                                                            |
| progress                             | 0                                                          |
| OS-EXT-STS:power_state               | 1                                                          |
| OS-EXT-AZ:availability_zone          | nova                                                       |
| config_drive                         |                                                            |
+--------------------------------------+------------------------------------------------------------+

[root@ctrl]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:50:56:85:3f:bb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:fe85:3fbb/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:50:56:85:58:e6 brd ff:ff:ff:ff:ff:ff
    inet 10.112.1.30/27 brd 10.112.1.31 scope global eth1
    inet6 fe80::250:56ff:fe85:58e6/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 00:50:56:85:37:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.2/22 brd 192.168.103.255 scope global eth2
    inet6 fe80::250:56ff:fe85:3756/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 0e:d1:a7:72:4a:75 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether ce:48:0e:5e:1e:4a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c0c:21ff:fee7:a88f/64 scope link 
       valid_lft forever preferred_lft forever
10: br-ex: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:50:56:85:3f:bb brd ff:ff:ff:ff:ff:ff
    inet 129.132.32.7/26 brd 129.132.32.63 scope global br-ex
    inet6 fe80::2050:61ff:fecd:8149/64 scope link 
       valid_lft forever preferred_lft forever
11: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a6:fe:6c:2e:15:4d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8cce:7eff:fe96:a516/64 scope link 
       valid_lft forever preferred_lft forever

[root@ctrl]# ip netns list
qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6
qdhcp-fa943fa6-6e44-46a8-a391-7f12a8249bf1

[root@ctrl]# ip netns exec qdhcp-fa943fa6-6e44-46a8-a391-7f12a8249bf1 ip a
15: tap0d9dbb61-f8: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:9c:4a:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.3/24 brd 10.1.0.255 scope global tap0d9dbb61-f8
    inet6 fe80::f816:3eff:fe9c:4ad8/64 scope link 
       valid_lft forever preferred_lft forever
16: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ip a
12: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
13: qr-ca4cb00d-22: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:0a:f5:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 brd 10.1.0.255 scope global qr-ca4cb00d-22
    inet6 fe80::f816:3eff:fe0a:f5c8/64 scope link 
       valid_lft forever preferred_lft forever
14: qg-5366ad09-03: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:06:00:6c brd ff:ff:ff:ff:ff:ff
    inet 172.31.116.7/24 brd 172.31.116.255 scope global qg-5366ad09-03
    inet 172.31.116.12/32 brd 172.31.116.12 scope global qg-5366ad09-03
    inet6 fe80::f816:3eff:fe06:6c/64 scope link 
       valid_lft forever preferred_lft forever

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ip r
172.31.116.0/24 dev qg-5366ad09-03  proto kernel  scope link  src 172.31.116.7 
10.1.0.0/24 dev qr-ca4cb00d-22  proto kernel  scope link  src 10.1.0.1 
default via 172.31.116.1 dev qg-5366ad09-03 

[root@ctrl]# iptables-save
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:09:42 2013
*mangle
:PREROUTING ACCEPT [156413:49499526]
:INPUT ACCEPT [156413:49499526]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [153794:50619929]
:POSTROUTING ACCEPT [153794:50619929]
:nova-api-POSTROUTING - [0:0]
-A POSTROUTING -j nova-api-POSTROUTING
COMMIT
# Completed on Tue Oct 15 16:09:42 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:09:42 2013
*nat
:PREROUTING ACCEPT [437:27722]
:POSTROUTING ACCEPT [875:53095]
:OUTPUT ACCEPT [875:53095]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-POSTROUTING - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
:neutron-openvswi-float-snat - [0:0]
:neutron-openvswi-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-POSTROUTING - [0:0]
:nova-api-PREROUTING - [0:0] 
:nova-api-float-snat - [0:0] 
:nova-api-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-openvswi-PREROUTING 
-A PREROUTING -j nova-api-PREROUTING 
-A POSTROUTING -j neutron-openvswi-POSTROUTING 
-A POSTROUTING -j neutron-postrouting-bottom 
-A POSTROUTING -j nova-api-POSTROUTING 
-A POSTROUTING -j nova-postrouting-bottom 
-A OUTPUT -j neutron-openvswi-OUTPUT 
-A OUTPUT -j nova-api-OUTPUT 
-A neutron-openvswi-snat -j neutron-openvswi-float-snat 
-A neutron-postrouting-bottom -j neutron-openvswi-snat 
-A nova-api-snat -j nova-api-float-snat 
-A nova-postrouting-bottom -j nova-api-snat 
COMMIT
# Completed on Tue Oct 15 16:09:42 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:09:42 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [154937:50867966]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT 
-A INPUT -j nova-api-INPUT 
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "accept incoming ESTABLISHED,RELATED" -j ACCEPT 
-A INPUT -p icmp -m comment --comment "accept incoming ICMP" -j ACCEPT 
-A INPUT -i lo -m comment --comment "accept incoming on loopback" -j ACCEPT 
-A INPUT -i eth1 -p gre -m state --state NEW -m comment --comment "accept incoming GRE" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m comment --comment "accept incoming SSH" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -m comment --comment "accept incoming HTTP" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 111 -m comment --comment "accept incoming RPC Portmapper" -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 111 -m comment --comment "accept incoming RPC Portmapper" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 123 -m comment --comment "drop incoming NTP" -j DROP 
-A INPUT -p udp -m state --state NEW -m udp --dport 123 -m comment --comment "drop incoming NTP" -j DROP 
-A INPUT -p udp -m state --state NEW -m udp --dport 137 -m comment --comment "drop incoming NetBIOS" -j DROP 
-A INPUT -p udp -m state --state NEW -m udp --dport 138 -m comment --comment "drop incoming NetBIOS" -j DROP 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -m comment --comment "accept incoming HTTPS" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2049 -m comment --comment "accept incoming NFSv4" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3260 -m comment --comment "accept incoming Cinder iSCSI" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -m comment --comment "accept incoming MySQL" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5000 -m comment --comment "accept incoming Keystone Identity API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -m comment --comment "accept incoming NRPE" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5672 -m comment --comment "accept incoming AMQP" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6080 -m comment --comment "accept incoming noVNC" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8773 -m comment --comment "accept incoming EC2 Compability API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8774 -m comment --comment "accept incoming Nova Compute API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8775 -m comment --comment "accept incoming Nova Metadata API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8776 -m comment --comment "accept incoming Cinder Volume API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9292 -m comment --comment "accept incoming Glance Image API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9696 -m comment --comment "accept incoming Quantum Networking API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 35357 -m comment --comment "accept incoming Keystone Identity API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 51234 -m comment --comment "accept incoming Func" -j ACCEPT 
-A INPUT -d 224.0.0.1/32 -m comment --comment "drop incoming multicast" -j DROP 
-A INPUT -j LOG --log-prefix "iptables INPUT       " 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j neutron-filter-top 
-A FORWARD -j neutron-openvswi-FORWARD 
-A FORWARD -j nova-filter-top 
-A FORWARD -j nova-api-FORWARD 
-A FORWARD -j LOG --log-prefix "iptables FORWARD     " 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
-A OUTPUT -j neutron-filter-top 
-A OUTPUT -j neutron-openvswi-OUTPUT 
-A OUTPUT -j nova-filter-top 
-A OUTPUT -j nova-api-OUTPUT 
-A neutron-filter-top -j neutron-openvswi-local 
-A neutron-openvswi-sg-fallback -j DROP 
-A nova-api-INPUT -d 129.132.32.7/32 -p tcp -m tcp --dport 8775 -j ACCEPT 
-A nova-filter-top -j nova-api-local 
COMMIT
# Completed on Tue Oct 15 16:09:42 2013

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 iptables-save
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:10:14 2013
*mangle
:PREROUTING ACCEPT [197:20582]
:INPUT ACCEPT [189:19620]
:FORWARD ACCEPT [6:504]
:OUTPUT ACCEPT [117:10974]
:POSTROUTING ACCEPT [121:11254]
COMMIT
# Completed on Tue Oct 15 16:10:14 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:10:14 2013
*nat
:PREROUTING ACCEPT [30:7634]
:POSTROUTING ACCEPT [6:436]
:OUTPUT ACCEPT [6:436]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING 
-A POSTROUTING -j neutron-l3-agent-POSTROUTING 
-A POSTROUTING -j neutron-postrouting-bottom 
-A OUTPUT -j neutron-l3-agent-OUTPUT 
-A neutron-l3-agent-OUTPUT -d 172.31.116.12/32 -j DNAT --to-destination 10.1.0.2 
-A neutron-l3-agent-POSTROUTING ! -i qg-5366ad09-03 ! -o qg-5366ad09-03 -m conntrack ! --ctstate DNAT -j ACCEPT 
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 
-A neutron-l3-agent-PREROUTING -d 172.31.116.12/32 -j DNAT --to-destination 10.1.0.2 
-A neutron-l3-agent-float-snat -s 10.1.0.2/32 -j SNAT --to-source 172.31.116.12 
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat 
-A neutron-l3-agent-snat -s 10.1.0.0/24 -j SNAT --to-source 172.31.116.7 
-A neutron-postrouting-bottom -j neutron-l3-agent-snat 
COMMIT
# Completed on Tue Oct 15 16:10:14 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:10:14 2013
*filter
:INPUT ACCEPT [189:19620]
:FORWARD ACCEPT [6:504]
:OUTPUT ACCEPT [116:10817]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
-A INPUT -j neutron-l3-agent-INPUT 
-A FORWARD -j neutron-filter-top 
-A FORWARD -j neutron-l3-agent-FORWARD 
-A OUTPUT -j neutron-filter-top 
-A OUTPUT -j neutron-l3-agent-OUTPUT 
-A neutron-filter-top -j neutron-l3-agent-local 
-A neutron-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT 
COMMIT
# Completed on Tue Oct 15 16:10:14 2013

[root@ctrl]# ping -c1 172.31.116.1
PING 172.31.116.1 (172.31.116.1) 56(84) bytes of data.
64 bytes from 172.31.116.1: icmp_seq=1 ttl=255 time=0.698 ms

--- 172.31.116.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.698/0.698/0.698/0.000 ms
[root@ctrl]# ping -c1 172.31.116.7
PING 172.31.116.7 (172.31.116.7) 56(84) bytes of data.

--- 172.31.116.7 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

[root@ctrl]# ping -c1 172.31.116.12
PING 172.31.116.12 (172.31.116.12) 56(84) bytes of data.

--- 172.31.116.12 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=0.075 ms

--- 10.1.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.075/0.075/0.075/0.000 ms
[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 10.1.0.2
PING 10.1.0.2 (10.1.0.2) 56(84) bytes of data.
64 bytes from 10.1.0.2: icmp_seq=1 ttl=64 time=1.21 ms

--- 10.1.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 1.218/1.218/1.218/0.000 ms
[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 172.31.116.1
PING 172.31.116.1 (172.31.116.1) 56(84) bytes of data.
From 172.31.116.7 icmp_seq=1 Destination Host Unreachable

--- 172.31.116.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 3001ms

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 172.31.116.7
PING 172.31.116.7 (172.31.116.7) 56(84) bytes of data.
64 bytes from 172.31.116.7: icmp_seq=1 ttl=64 time=0.062 ms

--- 172.31.116.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.062/0.062/0.062/0.000 ms
[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 172.31.116.12
PING 172.31.116.12 (172.31.116.12) 56(84) bytes of data.
64 bytes from 172.31.116.12: icmp_seq=1 ttl=64 time=1.18 ms

--- 172.31.116.12 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 1.185/1.185/1.185/0.000 ms

[root@instance ~]# ping -c1 10.1.0.1
PING 10.1.0.1 (10.1.0.1): 56 data bytes
64 bytes from 10.1.0.1: seq=0 ttl=64 time=0.819 ms

--- 10.1.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.819/0.819/0.819 ms

[root@instance ~]# ping -c1 10.1.0.2
PING 10.1.0.2 (10.1.0.2): 56 data bytes
64 bytes from 10.1.0.2: seq=0 ttl=64 time=0.073 ms

--- 10.1.0.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.073/0.073/0.073 ms

[root@instance ~]# ping -c1 172.31.116.1
PING 172.31.116.1 (172.31.116.1): 56 data bytes

--- 172.31.116.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

[root@instance ~]# ping -c1 172.31.116.7
PING 172.31.116.7 (172.31.116.7): 56 data bytes
64 bytes from 172.31.116.7: seq=0 ttl=64 time=0.859 ms

--- 172.31.116.7 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.859/0.859/0.859 ms

[root@instance ~]# ping -c1 172.31.116.12
PING 172.31.116.12 (172.31.116.12): 56 data bytes
64 bytes from 172.31.116.12: seq=0 ttl=63 time=1.449 ms

--- 172.31.116.12 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.449/1.449/1.449 ms

Floating IP not working (in-/egress from/to outside) [Neutron, OVS, GRE]

Trying to get Havana (rc1 from RDO) to work with Neutron using OVS and GRE tunnels. There's a single controller which also acts as the network node and there's currently two compute nodes. Trying to launch an instance, it gets a fixed IP which seems to work well. But then I assign a floating IP which is supposedly routed within our network. Now I can neither access our network from within the instance nor the other way around. I can't even ping the external gateway from within the instance and I can't ping the instance from the controller (and yes, I opened up the security group for all ICMP from 0.0.0.0/0).

Controller (aka ctrl below): - eth0 added to br-ex - br-ex has IP of 129.132.32.7 (a range that is not used anywhere else in the setup) - eth1 has the IP 10.112.1.30, which is the management network (over which the GRE tunnels go) - eth2 connects to our storage network and can be ignored in this scenario - Can ping Floating Gateway (external gateway) in default namespace - Can ping Floating Router and Floating IP in qrouter namespace - Can NOT ping Floating Gateway in qrouter namespace - Can NOT ping Floating Router or Floating IP in default namespace

Compute Nodes: - eth0 has IP of 10.112.1.{5,13}, which again is the management network (over which the GRE tunnels go)

Instance (aka instance below): - Private IP of 10.1.0.2 - Floating IP of 172.31.116.12 - Can ping itself on Private IP - Can ping Private Gateway - Can ping itself on Floating IP - Can ping Floating Router - Can NOT ping Floating Gateway

So what seems to sort of lack is a connection between the two namespaces (default and qrouter). But a tcpdump on eth0 reveals that if the instance tries to ping the Floating Gateway, there'a a arp request but there doesn't seem to be an answer. My guess would be because the network doesn't know where to send to answer to (i.e. the Floating IP is unknown to the network). Really no idea what I'm missing here.

Suspiciously, ALL ports on the Floating Network are shown as DOWN (while those on the Private Network are UP). I know there used to be a bug where they were shown down but actually worked, but this should be fixed and I think with the bug all ports on all networks showed as DOWN so not seeing that bug here.

Below, I'll list everything that I think could matter. Logs look clean, so leaving them away but happy to provide any additional information. Any help is much appreciated, thanks!

[root@ctrl]# neutron net-show eth_zurich
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 90696787-0c11-4031-bc99-8d1db2d24744 |
| name                      | eth_zurich                           |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 4                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 47e26025-f09a-4299-915e-6b2275fea59e |
| tenant_id                 | b6f22facd6bb43a5bb465097b53a087a     |
+---------------------------+--------------------------------------+

[root@ctrl]# neutron subnet-show private_v4
+------------------+---------------------------------------------------+
| Field            | Value                                             |
+------------------+---------------------------------------------------+
| allocation_pools | {"start": "172.31.116.7", "end": "172.31.116.59"} |
| cidr             | 172.31.116.0/24                                   |
| dns_nameservers  |                                                   |
| enable_dhcp      | False                                             |
| gateway_ip       | 172.31.116.1                                      |
| host_routes      |                                                   |
| id               | 47e26025-f09a-4299-915e-6b2275fea59e              |
| ip_version       | 4                                                 |
| name             | private_v4                                        |
| network_id       | 90696787-0c11-4031-bc99-8d1db2d24744              |
| tenant_id        | 76896fef704649c19d4fc921e9e2e680                  |
+------------------+---------------------------------------------------+

[root@ctrl]# neutron router-show portal
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "90696787-0c11-4031-bc99-8d1db2d24744", "enable_snat": true} |
| id                    | 0ca86e83-80fe-403e-83e2-d4652985cee6                                        |
| name                  | portal                                                                      |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | 76896fef704649c19d4fc921e9e2e680                                            |
+-----------------------+-----------------------------------------------------------------------------+

[root@ctrl]# neutron port-list
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                            |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 0d9dbb61-f8bf-4741-b94d-fcd835245747 |      | fa:16:3e:9c:4a:d8 | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.3"}      |
| 5366ad09-03ce-4426-8664-57cba7601833 |      | fa:16:3e:06:00:6c | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.7"}  |
| b9587621-f1d3-4268-ba5b-3a77c6f94037 |      | fa:16:3e:cb:47:8b | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.11"} |
| ca4cb00d-22b5-4973-b36f-63d0426cdc52 |      | fa:16:3e:0a:f5:c8 | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.1"}      |
| cf2eaeb4-5933-4d02-a8c1-969420e12632 |      | fa:16:3e:59:40:53 | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.12"} |
| d447ef74-70d2-4f13-a584-a6ac0c9d23d8 |      | fa:16:3e:68:6a:89 | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.2"}      |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+

[root@ctrl]# neutron port-show d447ef74-70d2-4f13-a584-a6ac0c9d23d8
+-----------------------+---------------------------------------------------------------------------------+
| Field                 | Value                                                                           |
+-----------------------+---------------------------------------------------------------------------------+
| admin_state_up        | True                                                                            |
| allowed_address_pairs |                                                                                 |
| binding:capabilities  | {"port_filter": true}                                                           |
| binding:host_id       | node05.ethz.lan                                                                 |
| binding:vif_type      | ovs                                                                             |
| device_id             | fd10e420-bab2-4209-95b7-7c18918ef48e                                            |
| device_owner          | compute:nova                                                                    |
| extra_dhcp_opts       |                                                                                 |
| fixed_ips             | {"subnet_id": "2a7bbbe5-db9c-486d-abe5-515961a56b23", "ip_address": "10.1.0.2"} |
| id                    | d447ef74-70d2-4f13-a584-a6ac0c9d23d8                                            |
| mac_address           | fa:16:3e:68:6a:89                                                               |
| name                  |                                                                                 |
| network_id            | fa943fa6-6e44-46a8-a391-7f12a8249bf1                                            |
| security_groups       | 35953a34-2ebc-4f12-a3c7-3aeac6de2dc2                                            |
| status                | ACTIVE                                                                          |
| tenant_id             | 76896fef704649c19d4fc921e9e2e680                                                |
+-----------------------+---------------------------------------------------------------------------------+

[root@ctrl]# neutron port-show cf2eaeb4-5933-4d02-a8c1-969420e12632
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value                                                                                |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                 |
| allowed_address_pairs |                                                                                      |
| binding:capabilities  | {"port_filter": true}                                                                |
| binding:host_id       |                                                                                      |
| binding:vif_type      | ovs                                                                                  |
| device_id             | ae505a31-584d-465c-b922-69da4bc5f059                                                 |
| device_owner          | network:floatingip                                                                   |
| extra_dhcp_opts       |                                                                                      |
| fixed_ips             | {"subnet_id": "47e26025-f09a-4299-915e-6b2275fea59e", "ip_address": "172.31.116.12"} |
| id                    | cf2eaeb4-5933-4d02-a8c1-969420e12632                                                 |
| mac_address           | fa:16:3e:59:40:53                                                                    |
| name                  |                                                                                      |
| network_id            | 90696787-0c11-4031-bc99-8d1db2d24744                                                 |
| security_groups       |                                                                                      |
| status                | DOWN                                                                                 |
| tenant_id             |                                                                                      |
+-----------------------+--------------------------------------------------------------------------------------+

[root@ctrl]# neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 46731d68-3843-4642-b093-06816ba0246c |                  | 172.31.116.11       |                                      |
| ae505a31-584d-465c-b922-69da4bc5f059 | 10.1.0.2         | 172.31.116.12       | d447ef74-70d2-4f13-a584-a6ac0c9d23d8 |
+--------------------------------------+------------------+---------------------+--------------------------------------+

[root@ctrl]# neutron floatingip-show ae505a31-584d-465c-b922-69da4bc5f059
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.1.0.2                             |
| floating_ip_address | 172.31.116.12                        |
| floating_network_id | 90696787-0c11-4031-bc99-8d1db2d24744 |
| id                  | ae505a31-584d-465c-b922-69da4bc5f059 |
| port_id             | d447ef74-70d2-4f13-a584-a6ac0c9d23d8 |
| router_id           | 0ca86e83-80fe-403e-83e2-d4652985cee6 |
| tenant_id           | 76896fef704649c19d4fc921e9e2e680     |
+---------------------+--------------------------------------+

[root@ctrl]# nova list
+--------------------------------------+------+--------+------------+-------------+-----------------------------------+
| ID                                   | Name | Status | Task State | Power State | Networks                          |
+--------------------------------------+------+--------+------------+-------------+-----------------------------------+
| fd10e420-bab2-4209-95b7-7c18918ef48e | CT   | ACTIVE | None       | Running     | instances=10.1.0.2, 172.31.116.12 |
+--------------------------------------+------+--------+------------+-------------+-----------------------------------+

[root@ctrl]# nova show CT
+--------------------------------------+------------------------------------------------------------+
| Property                             | Value                                                      |
+--------------------------------------+------------------------------------------------------------+
| status                               | ACTIVE                                                     |
| updated                              | 2013-10-15T13:48:37Z                                       |
| OS-EXT-STS:task_state                | None                                                       |
| OS-EXT-SRV-ATTR:host                 | node05.ethz.lan                                            |
| key_name                             | None                                                       |
| image                                | Cirros 0.3.1 x86_64 (cfdfdaba-6d71-4be5-a17c-4b67ed18b130) |
| hostId                               | 2147b671c0f53d0bccbf737b305ac02ce48fe3aee9e4a2223df59fdd   |
| OS-EXT-STS:vm_state                  | active                                                     |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000000b                                          |
| OS-SRV-USG:launched_at               | 2013-10-15T13:48:37.000000                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node05.ethz.lan                                            |
| flavor                               | m1.tiny (1)                                                |
| id                                   | fd10e420-bab2-4209-95b7-7c18918ef48e                       |
| security_groups                      | [{u'name': u'default'}]                                    |
| OS-SRV-USG:terminated_at             | None                                                       |
| instances network                    | 10.1.0.2, 172.31.116.12                                    |
| user_id                              | e18119c0273d43b1b695094b746d3814                           |
| name                                 | CT                                                         |
| created                              | 2013-10-15T13:48:16Z                                       |
| tenant_id                            | 76896fef704649c19d4fc921e9e2e680                           |
| OS-DCF:diskConfig                    | MANUAL                                                     |
| metadata                             | {}                                                         |
| os-extended-volumes:volumes_attached | []                                                         |
| accessIPv4                           |                                                            |
| accessIPv6                           |                                                            |
| progress                             | 0                                                          |
| OS-EXT-STS:power_state               | 1                                                          |
| OS-EXT-AZ:availability_zone          | nova                                                       |
| config_drive                         |                                                            |
+--------------------------------------+------------------------------------------------------------+

[root@ctrl]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:50:56:85:3f:bb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:fe85:3fbb/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:50:56:85:58:e6 brd ff:ff:ff:ff:ff:ff
    inet 10.112.1.30/27 brd 10.112.1.31 scope global eth1
    inet6 fe80::250:56ff:fe85:58e6/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 00:50:56:85:37:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.2/22 brd 192.168.103.255 scope global eth2
    inet6 fe80::250:56ff:fe85:3756/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 0e:d1:a7:72:4a:75 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether ce:48:0e:5e:1e:4a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c0c:21ff:fee7:a88f/64 scope link 
       valid_lft forever preferred_lft forever
10: br-ex: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:50:56:85:3f:bb brd ff:ff:ff:ff:ff:ff
    inet 129.132.32.7/26 brd 129.132.32.63 scope global br-ex
    inet6 fe80::2050:61ff:fecd:8149/64 scope link 
       valid_lft forever preferred_lft forever
11: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a6:fe:6c:2e:15:4d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8cce:7eff:fe96:a516/64 scope link 
       valid_lft forever preferred_lft forever

[root@ctrl]# ip netns list
qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6
qdhcp-fa943fa6-6e44-46a8-a391-7f12a8249bf1

[root@ctrl]# ip netns exec qdhcp-fa943fa6-6e44-46a8-a391-7f12a8249bf1 ip a
15: tap0d9dbb61-f8: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:9c:4a:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.3/24 brd 10.1.0.255 scope global tap0d9dbb61-f8
    inet6 fe80::f816:3eff:fe9c:4ad8/64 scope link 
       valid_lft forever preferred_lft forever
16: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ip a
12: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
13: qr-ca4cb00d-22: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:0a:f5:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 brd 10.1.0.255 scope global qr-ca4cb00d-22
    inet6 fe80::f816:3eff:fe0a:f5c8/64 scope link 
       valid_lft forever preferred_lft forever
14: qg-5366ad09-03: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:06:00:6c brd ff:ff:ff:ff:ff:ff
    inet 172.31.116.7/24 brd 172.31.116.255 scope global qg-5366ad09-03
    inet 172.31.116.12/32 brd 172.31.116.12 scope global qg-5366ad09-03
    inet6 fe80::f816:3eff:fe06:6c/64 scope link 
       valid_lft forever preferred_lft forever

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ip r
172.31.116.0/24 dev qg-5366ad09-03  proto kernel  scope link  src 172.31.116.7 
10.1.0.0/24 dev qr-ca4cb00d-22  proto kernel  scope link  src 10.1.0.1 
default via 172.31.116.1 dev qg-5366ad09-03 

[root@ctrl]# iptables-save
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:09:42 2013
*mangle
:PREROUTING ACCEPT [156413:49499526]
:INPUT ACCEPT [156413:49499526]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [153794:50619929]
:POSTROUTING ACCEPT [153794:50619929]
:nova-api-POSTROUTING - [0:0]
-A POSTROUTING -j nova-api-POSTROUTING
COMMIT
# Completed on Tue Oct 15 16:09:42 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:09:42 2013
*nat
:PREROUTING ACCEPT [437:27722]
:POSTROUTING ACCEPT [875:53095]
:OUTPUT ACCEPT [875:53095]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-POSTROUTING - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
:neutron-openvswi-float-snat - [0:0]
:neutron-openvswi-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-POSTROUTING - [0:0]
:nova-api-PREROUTING - [0:0] 
:nova-api-float-snat - [0:0] 
:nova-api-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-openvswi-PREROUTING 
-A PREROUTING -j nova-api-PREROUTING 
-A POSTROUTING -j neutron-openvswi-POSTROUTING 
-A POSTROUTING -j neutron-postrouting-bottom 
-A POSTROUTING -j nova-api-POSTROUTING 
-A POSTROUTING -j nova-postrouting-bottom 
-A OUTPUT -j neutron-openvswi-OUTPUT 
-A OUTPUT -j nova-api-OUTPUT 
-A neutron-openvswi-snat -j neutron-openvswi-float-snat 
-A neutron-postrouting-bottom -j neutron-openvswi-snat 
-A nova-api-snat -j nova-api-float-snat 
-A nova-postrouting-bottom -j nova-api-snat 
COMMIT
# Completed on Tue Oct 15 16:09:42 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:09:42 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [154937:50867966]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT 
-A INPUT -j nova-api-INPUT 
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "accept incoming ESTABLISHED,RELATED" -j ACCEPT 
-A INPUT -p icmp -m comment --comment "accept incoming ICMP" -j ACCEPT 
-A INPUT -i lo -m comment --comment "accept incoming on loopback" -j ACCEPT 
-A INPUT -i eth1 -p gre -m state --state NEW -m comment --comment "accept incoming GRE" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m comment --comment "accept incoming SSH" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -m comment --comment "accept incoming HTTP" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 111 -m comment --comment "accept incoming RPC Portmapper" -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 111 -m comment --comment "accept incoming RPC Portmapper" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 123 -m comment --comment "drop incoming NTP" -j DROP 
-A INPUT -p udp -m state --state NEW -m udp --dport 123 -m comment --comment "drop incoming NTP" -j DROP 
-A INPUT -p udp -m state --state NEW -m udp --dport 137 -m comment --comment "drop incoming NetBIOS" -j DROP 
-A INPUT -p udp -m state --state NEW -m udp --dport 138 -m comment --comment "drop incoming NetBIOS" -j DROP 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -m comment --comment "accept incoming HTTPS" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2049 -m comment --comment "accept incoming NFSv4" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3260 -m comment --comment "accept incoming Cinder iSCSI" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -m comment --comment "accept incoming MySQL" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5000 -m comment --comment "accept incoming Keystone Identity API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -m comment --comment "accept incoming NRPE" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5672 -m comment --comment "accept incoming AMQP" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6080 -m comment --comment "accept incoming noVNC" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8773 -m comment --comment "accept incoming EC2 Compability API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8774 -m comment --comment "accept incoming Nova Compute API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8775 -m comment --comment "accept incoming Nova Metadata API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8776 -m comment --comment "accept incoming Cinder Volume API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9292 -m comment --comment "accept incoming Glance Image API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9696 -m comment --comment "accept incoming Quantum Networking API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 35357 -m comment --comment "accept incoming Keystone Identity API" -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 51234 -m comment --comment "accept incoming Func" -j ACCEPT 
-A INPUT -d 224.0.0.1/32 -m comment --comment "drop incoming multicast" -j DROP 
-A INPUT -j LOG --log-prefix "iptables INPUT       " 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j neutron-filter-top 
-A FORWARD -j neutron-openvswi-FORWARD 
-A FORWARD -j nova-filter-top 
-A FORWARD -j nova-api-FORWARD 
-A FORWARD -j LOG --log-prefix "iptables FORWARD     " 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
-A OUTPUT -j neutron-filter-top 
-A OUTPUT -j neutron-openvswi-OUTPUT 
-A OUTPUT -j nova-filter-top 
-A OUTPUT -j nova-api-OUTPUT 
-A neutron-filter-top -j neutron-openvswi-local 
-A neutron-openvswi-sg-fallback -j DROP 
-A nova-api-INPUT -d 129.132.32.7/32 -p tcp -m tcp --dport 8775 -j ACCEPT 
-A nova-filter-top -j nova-api-local 
COMMIT
# Completed on Tue Oct 15 16:09:42 2013

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 iptables-save
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:10:14 2013
*mangle
:PREROUTING ACCEPT [197:20582]
:INPUT ACCEPT [189:19620]
:FORWARD ACCEPT [6:504]
:OUTPUT ACCEPT [117:10974]
:POSTROUTING ACCEPT [121:11254]
COMMIT
# Completed on Tue Oct 15 16:10:14 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:10:14 2013
*nat
:PREROUTING ACCEPT [30:7634]
:POSTROUTING ACCEPT [6:436]
:OUTPUT ACCEPT [6:436]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING 
-A POSTROUTING -j neutron-l3-agent-POSTROUTING 
-A POSTROUTING -j neutron-postrouting-bottom 
-A OUTPUT -j neutron-l3-agent-OUTPUT 
-A neutron-l3-agent-OUTPUT -d 172.31.116.12/32 -j DNAT --to-destination 10.1.0.2 
-A neutron-l3-agent-POSTROUTING ! -i qg-5366ad09-03 ! -o qg-5366ad09-03 -m conntrack ! --ctstate DNAT -j ACCEPT 
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 
-A neutron-l3-agent-PREROUTING -d 172.31.116.12/32 -j DNAT --to-destination 10.1.0.2 
-A neutron-l3-agent-float-snat -s 10.1.0.2/32 -j SNAT --to-source 172.31.116.12 
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat 
-A neutron-l3-agent-snat -s 10.1.0.0/24 -j SNAT --to-source 172.31.116.7 
-A neutron-postrouting-bottom -j neutron-l3-agent-snat 
COMMIT
# Completed on Tue Oct 15 16:10:14 2013
# Generated by iptables-save v1.4.7 on Tue Oct 15 16:10:14 2013
*filter
:INPUT ACCEPT [189:19620]
:FORWARD ACCEPT [6:504]
:OUTPUT ACCEPT [116:10817]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
-A INPUT -j neutron-l3-agent-INPUT 
-A FORWARD -j neutron-filter-top 
-A FORWARD -j neutron-l3-agent-FORWARD 
-A OUTPUT -j neutron-filter-top 
-A OUTPUT -j neutron-l3-agent-OUTPUT 
-A neutron-filter-top -j neutron-l3-agent-local 
-A neutron-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT 
COMMIT
# Completed on Tue Oct 15 16:10:14 2013

[root@ctrl]# ping -c1 172.31.116.1
PING 172.31.116.1 (172.31.116.1) 56(84) bytes of data.
64 bytes from 172.31.116.1: icmp_seq=1 ttl=255 time=0.698 ms

--- 172.31.116.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.698/0.698/0.698/0.000 ms
[root@ctrl]# ping -c1 172.31.116.7
PING 172.31.116.7 (172.31.116.7) 56(84) bytes of data.

--- 172.31.116.7 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

[root@ctrl]# ping -c1 172.31.116.12
PING 172.31.116.12 (172.31.116.12) 56(84) bytes of data.

--- 172.31.116.12 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=0.075 ms

--- 10.1.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.075/0.075/0.075/0.000 ms
[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 10.1.0.2
PING 10.1.0.2 (10.1.0.2) 56(84) bytes of data.
64 bytes from 10.1.0.2: icmp_seq=1 ttl=64 time=1.21 ms

--- 10.1.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 1.218/1.218/1.218/0.000 ms
[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 172.31.116.1
PING 172.31.116.1 (172.31.116.1) 56(84) bytes of data.
From 172.31.116.7 icmp_seq=1 Destination Host Unreachable

--- 172.31.116.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 3001ms

[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 172.31.116.7
PING 172.31.116.7 (172.31.116.7) 56(84) bytes of data.
64 bytes from 172.31.116.7: icmp_seq=1 ttl=64 time=0.062 ms

--- 172.31.116.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.062/0.062/0.062/0.000 ms
[root@ctrl]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 ping -c1 172.31.116.12
PING 172.31.116.12 (172.31.116.12) 56(84) bytes of data.
64 bytes from 172.31.116.12: icmp_seq=1 ttl=64 time=1.18 ms

--- 172.31.116.12 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 1.185/1.185/1.185/0.000 ms

[root@instance ~]# ping -c1 10.1.0.1
PING 10.1.0.1 (10.1.0.1): 56 data bytes
64 bytes from 10.1.0.1: seq=0 ttl=64 time=0.819 ms

--- 10.1.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.819/0.819/0.819 ms

[root@instance ~]# ping -c1 10.1.0.2
PING 10.1.0.2 (10.1.0.2): 56 data bytes
64 bytes from 10.1.0.2: seq=0 ttl=64 time=0.073 ms

--- 10.1.0.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.073/0.073/0.073 ms

[root@instance ~]# ping -c1 172.31.116.1
PING 172.31.116.1 (172.31.116.1): 56 data bytes

--- 172.31.116.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

[root@instance ~]# ping -c1 172.31.116.7
PING 172.31.116.7 (172.31.116.7): 56 data bytes
64 bytes from 172.31.116.7: seq=0 ttl=64 time=0.859 ms

--- 172.31.116.7 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.859/0.859/0.859 ms

[root@instance ~]# ping -c1 172.31.116.12
PING 172.31.116.12 (172.31.116.12): 56 data bytes
64 bytes from 172.31.116.12: seq=0 ttl=63 time=1.449 ms

--- 172.31.116.12 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.449/1.449/1.449 ms

UPDATE (answers for Darragh): The controller is a VMware based VM, the compute nodes are HW-based. VMware is running on the same kind of blades as the nodes in the same chassis with the same switches, routers, etc.

129.132.32.7/26 is the external connectivity but I can take it away from br-ex and add a dedicated interface. Tried that but with the same results. I can also use an IP from the 172.31.116.0/24 network instead and still get the same results. Both networks are available on the same interface(s) and are configured on the same router.

Here's the tcpdump from another VMware based VM ("mgmt") that is basically sitting next to the controller, so you see it's reaching the network. That VM does have an IP on the 129.132.32.0/26 but not on the 172.31.116.0/24 network. Pinging the gateway from within the instance, the network sees only just this:

[root@mgmt ~]# tcpdump -i eth3 -n > tcpdump; grep -v ssh tcpdump
10:10:46.264958 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:48.265144 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:49.265141 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:50.265143 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:51.265765 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:52.265769 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:53.265769 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46
10:10:55.265769 ARP, Request who-has 172.31.116.1 tell 172.31.116.12, length 46

I also did some ARP pings (not reflected in the tcpdump).

[root@mgmt ~]# arping -I eth3 -c 1 172.31.116.1
ARPING 172.31.116.1 from 129.132.32.10 eth3
Unicast reply from 172.31.116.1 [04:C5:A4:E3:FA:42]  1.158ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

[root@mgmt ~]# arping -I eth3 -c 1 129.132.32.1
ARPING 129.132.32.1 from 129.132.32.10 eth3
Unicast reply from 129.132.32.1 [04:C5:A4:E3:FA:42]  1.309ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

[root@ctrl-stg ~]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 arping -I qg-5366ad09-03 -c 1 172.31.116.1
sysfs read broadcast value: No such file or directory
falling back to default broadcast value
ARPING 172.31.116.1 from 172.31.116.7 qg-5366ad09-03
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

[root@ctrl-stg ~]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 arping -I qg-5366ad09-03 -c 1 129.132.32.1
sysfs read broadcast value: No such file or directory
falling back to default broadcast value
ARPING 129.132.32.1 from 172.31.116.7 qg-5366ad09-03
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

[root@ctrl-stg ~]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 arping -I qr-ca4cb00d-22 -c 1 172.31.116.1
sysfs read broadcast value: No such file or directory
falling back to default broadcast value
ARPING 172.31.116.1 from 10.1.0.1 qr-ca4cb00d-22
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

[root@ctrl-stg ~]# ip netns exec qrouter-0ca86e83-80fe-403e-83e2-d4652985cee6 arping -I qr-ca4cb00d-22 -c 1 129.132.32.1
sysfs read broadcast value: No such file or directory
falling back to default broadcast value
ARPING 129.132.32.1 from 10.1.0.1 qr-ca4cb00d-22
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

Let me know if you need tcpdump of some (or all) of those, too.