Revision history [back]

click to hide/show revision 1
initial version

can't ping external gateway or floating IP's from within router namespace

I just finished setting up an OpenStack packstack deployment for testing purposes. I was able to launch an instance and connect to it using ssh on the internal IP address. My router connects my external and internal networks and my security group is wide open. I've setup my bridge interface so that it links to eth0 which is my external network.

[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ssh 192.168.10.152 -l centos -i ~/.ssh/my-key
[centos@r20 ~]$

internal ping works:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ping 192.168.10.152    
PING 192.168.10.152 (192.168.10.152) 56(84) bytes of data.
64 bytes from 192.168.10.152: icmp_seq=1 ttl=64 time=1.66 ms
64 bytes from 192.168.10.152: icmp_seq=2 ttl=64 time=0.721 ms
64 bytes from 192.168.10.152: icmp_seq=3 ttl=64 time=0.697 ms
64 bytes from 192.168.10.152: icmp_seq=4 ttl=64 time=0.662 ms

external ping doesn't:

[root@ip-172-31-15-114 ~(keystone_admin)]# ping 172.31.0.10
PING 172.31.0.10 (172.31.0.10) 56(84) bytes of data.
From 172.31.15.114 icmp_seq=1 Destination Host Unreachable
From 172.31.15.114 icmp_seq=2 Destination Host Unreachable
From 172.31.15.114 icmp_seq=3 Destination Host Unreachable
From 172.31.15.114 icmp_seq=4 Destination Host Unreachable
^C
--- 172.31.0.10 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3000ms
pipe 4
[root@ip-172-31-15-114 ~(keystone_admin)]#

unable to ping Google from within instance:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ssh 192.168.10.152 -l centos -i ~/.ssh/my-key
Last login: Sat Mar  9 21:36:08 2019 from gateway
[centos@r20 ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 172.31.0.10 icmp_seq=1 Destination Host Unreachable
From 172.31.0.10 icmp_seq=2 Destination Host Unreachable
From 172.31.0.10 icmp_seq=3 Destination Host Unreachable
From 172.31.0.10 icmp_seq=4 Destination Host Unreachable
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3002ms
pipe 4
[centos@r20 ~]$ ip route
default via 192.168.10.1 dev eth0 
169.254.0.0/16 dev eth0 scope link metric 1002 
169.254.169.254 via 192.168.10.1 dev eth0 proto static 
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.152

pinging my gateway from my namespace doesn't work:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip route
default via 172.31.0.1 dev br-ex 
169.254.0.0/16 dev eth0 scope link metric 1002 
169.254.0.0/16 dev br-ex scope link metric 1006 
172.31.0.0/20 dev br-ex proto kernel scope link src 172.31.15.114 
[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ping 172.31.0.1
PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
From 172.31.0.31 icmp_seq=1 Destination Host Unreachable
From 172.31.0.31 icmp_seq=2 Destination Host Unreachable
From 172.31.0.31 icmp_seq=3 Destination Host Unreachable
From 172.31.0.31 icmp_seq=4 Destination Host Unreachable
^C
--- 172.31.0.1 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4000ms
pipe 4
[root@ip-172-31-15-114 ~(keystone_admin)]#

ip a output:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 02:71:3c:33:4d:48 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::71:3cff:fe33:4d48/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ae:80:6a:78:86:40 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 02:71:3c:33:4d:48 brd ff:ff:ff:ff:ff:ff
    inet 172.31.15.114/20 brd 172.31.15.255 scope global dynamic br-ex
       valid_lft 2229sec preferred_lft 2229sec
    inet6 fe80::71:3cff:fe33:4d48/64 scope link 
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 0e:f8:72:e6:26:4a brd ff:ff:ff:ff:ff:ff
8: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:24:bb:cf:29:41 brd ff:ff:ff:ff:ff:ff
16: qbrda5146c5-ef: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether b6:6c:bb:01:ad:77 brd ff:ff:ff:ff:ff:ff
17: qvoda5146c5-ef@qvbda5146c5-ef: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 9a:96:32:d8:48:f0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9896:32ff:fed8:48f0/64 scope link 
       valid_lft forever preferred_lft forever
18: qvbda5146c5-ef@qvoda5146c5-ef: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc noqueue master qbrda5146c5-ef state UP group default qlen 1000
    link/ether b6:6c:bb:01:ad:77 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b46c:bbff:fe01:ad77/64 scope link 
       valid_lft forever preferred_lft forever
19: tapda5146c5-ef: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbrda5146c5-ef state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:17:23:be brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe17:23be/64 scope link 
       valid_lft forever preferred_lft forever

I'm running Openstack Rocky on CentOS 7.6 with selinux set to permissive in an AWS ec2 instance (t2.2xlarge).

What are some additional troubleshooting steps I can take to identify the problem?

can't ping external gateway or floating IP's from within router namespace

I just finished setting up an OpenStack packstack deployment for testing purposes. I was able to launch an instance and connect to it using ssh on the internal IP address. My router connects my external and internal networks and my security group is wide open. I've setup my bridge interface so that it links to eth0 which is my external network.

[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ssh 192.168.10.152 -l centos -i ~/.ssh/my-key
[centos@r20 ~]$

internal ping works:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ping 192.168.10.152    
PING 192.168.10.152 (192.168.10.152) 56(84) bytes of data.
64 bytes from 192.168.10.152: icmp_seq=1 ttl=64 time=1.66 ms
64 bytes from 192.168.10.152: icmp_seq=2 ttl=64 time=0.721 ms
64 bytes from 192.168.10.152: icmp_seq=3 ttl=64 time=0.697 ms
64 bytes from 192.168.10.152: icmp_seq=4 ttl=64 time=0.662 ms

external ping doesn't:

[root@ip-172-31-15-114 ~(keystone_admin)]# ping 172.31.0.10
PING 172.31.0.10 (172.31.0.10) 56(84) bytes of data.
From 172.31.15.114 icmp_seq=1 Destination Host Unreachable
From 172.31.15.114 icmp_seq=2 Destination Host Unreachable
From 172.31.15.114 icmp_seq=3 Destination Host Unreachable
From 172.31.15.114 icmp_seq=4 Destination Host Unreachable
^C
--- 172.31.0.10 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3000ms
pipe 4
[root@ip-172-31-15-114 ~(keystone_admin)]#

unable to ping Google from within instance:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ssh 192.168.10.152 -l centos -i ~/.ssh/my-key
Last login: Sat Mar  9 21:36:08 2019 from gateway
[centos@r20 ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 172.31.0.10 icmp_seq=1 Destination Host Unreachable
From 172.31.0.10 icmp_seq=2 Destination Host Unreachable
From 172.31.0.10 icmp_seq=3 Destination Host Unreachable
From 172.31.0.10 icmp_seq=4 Destination Host Unreachable
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3002ms
pipe 4
[centos@r20 ~]$ ip route
default via 192.168.10.1 dev eth0 
169.254.0.0/16 dev eth0 scope link metric 1002 
169.254.169.254 via 192.168.10.1 dev eth0 proto static 
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.152

pinging my gateway from my namespace doesn't work:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip route
default via 172.31.0.1 dev br-ex 
169.254.0.0/16 dev eth0 scope link metric 1002 
169.254.0.0/16 dev br-ex scope link metric 1006 
172.31.0.0/20 dev br-ex proto kernel scope link src 172.31.15.114 
[root@ip-172-31-15-114 ~(keystone_admin)]# ip netns exec qrouter-7808ea77-12b3-432a-be6d-f85f2b980577 ping 172.31.0.1
PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
From 172.31.0.31 icmp_seq=1 Destination Host Unreachable
From 172.31.0.31 icmp_seq=2 Destination Host Unreachable
From 172.31.0.31 icmp_seq=3 Destination Host Unreachable
From 172.31.0.31 icmp_seq=4 Destination Host Unreachable
^C
--- 172.31.0.1 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4000ms
pipe 4
[root@ip-172-31-15-114 ~(keystone_admin)]#

ip a output:

[root@ip-172-31-15-114 ~(keystone_admin)]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 02:71:3c:33:4d:48 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::71:3cff:fe33:4d48/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ae:80:6a:78:86:40 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 02:71:3c:33:4d:48 brd ff:ff:ff:ff:ff:ff
    inet 172.31.15.114/20 brd 172.31.15.255 scope global dynamic br-ex
       valid_lft 2229sec preferred_lft 2229sec
    inet6 fe80::71:3cff:fe33:4d48/64 scope link 
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 0e:f8:72:e6:26:4a brd ff:ff:ff:ff:ff:ff
8: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:24:bb:cf:29:41 brd ff:ff:ff:ff:ff:ff
16: qbrda5146c5-ef: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether b6:6c:bb:01:ad:77 brd ff:ff:ff:ff:ff:ff
17: qvoda5146c5-ef@qvbda5146c5-ef: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 9a:96:32:d8:48:f0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9896:32ff:fed8:48f0/64 scope link 
       valid_lft forever preferred_lft forever
18: qvbda5146c5-ef@qvoda5146c5-ef: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc noqueue master qbrda5146c5-ef state UP group default qlen 1000
    link/ether b6:6c:bb:01:ad:77 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b46c:bbff:fe01:ad77/64 scope link 
       valid_lft forever preferred_lft forever
19: tapda5146c5-ef: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbrda5146c5-ef state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:17:23:be brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe17:23be/64 scope link 
       valid_lft forever preferred_lft forever

I'm running Openstack Rocky on CentOS 7.6 with selinux set to permissive in an AWS ec2 instance (t2.2xlarge).

What are some additional troubleshooting steps I can take to identify the problem?

Edit 1: Hi Stef, thank you for your reply. Here is the output of the command you requested:

**[root@ip-172-31-15-114 ~(keystone_admin)]# openstack network show external
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2019-03-09T20:13:33Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 46e46a94-5d68-4ce5-af28-ce67ca5a7baa |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1450                                 |
| name                      | external                             |
| port_security_enabled     | True                                 |
| project_id                | ead9ce3b26a64152ab075afc3b3c9361     |
| provider:network_type     | vxlan                                |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 16                                   |
| qos_policy_id             | None                                 |
| revision_number           | 6                                    |
| router:external           | External                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 10aa47d9-8112-4141-9b35-be51c7ee08a7 |
| tags                      |                                      |
| updated_at                | 2019-03-09T20:13:46Z                 |
+---------------------------+--------------------------------------+**