Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Neutron: Can't ping internet from instance and router namespace

Hi,

I have been trying to configure an openstack kilo environment based on 4 virtual machines with the following network configuration:

  • Controller01: external ip 192.168.1.135, management ip 192.168.50.135, tunnel ip 192.168.60.135
  • Compute01: external ip 192.168.1.136, management ip 192.168.50.136, tunnel ip 192.168.60.136
  • Compute02: external ip 192.168.1.140, management ip 192.168.50.140, tunnel ip 192.168.60.140
  • Network01: no external ip as per installation guide, management ip 192.168.50.137, tunnel ip 192.168.60.137. The external interface is eth1

My issue is that I can't ping the internet or outside world from the instances or from the router namespace. My instance subnet is 10.10.10.0. My qrouter has 2 interfaces: 10.10.10.1 and 192.168.1.160.

From an instance, I can ping 10.10.10.1, 192.168.1.160 but not my default gateway, which is 192.168.1.1 or anything else on the 192.168.1.x network. From the instance, I can also ping another instance and from the namespace of the qrouter, I can ping the instance as well.

Here are some information from Network01:

root@network01:~# ip netns
qdhcp-5c2bd816-81b9-4c42-acf5-3103fdceabed
qrouter-88baa567-97f2-48be-883c-a7ae389a60c5

root@network01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:40:7d:2a brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe40:7d2a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether 08:00:27:6c:6f:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a00:27ff:fe6c:6f1a/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:4c:b3:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.137/24 brd 192.168.50.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe4c:b363/64 scope link 
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:7f:46:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.60.137/24 brd 192.168.60.255 scope global eth3
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7f:46a3/64 scope link 
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 7a:e3:56:b6:03:39 brd ff:ff:ff:ff:ff:ff
7: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 08:00:27:6c:6f:1a brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 7a:56:2d:90:8f:4b brd ff:ff:ff:ff:ff:ff
12: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether fe:8e:43:08:49:4e brd ff:ff:ff:ff:ff:ff

root@network01:~# ip r
default via 10.0.2.2 dev eth0 
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.15 
192.168.50.0/24 dev eth2  proto kernel  scope link  src 192.168.50.137 
192.168.60.0/24 dev eth3  proto kernel  scope link  src 192.168.60.137

root@network01:~# ip netns exec qrouter-88baa567-97f2-48be-883c-a7ae389a60c5 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: qr-3fa6c9e7-5f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:5b:b9:4b brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 brd 10.10.10.255 scope global qr-3fa6c9e7-5f
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe5b:b94b/64 scope link 
       valid_lft forever preferred_lft forever
15: qg-91e2a788-b2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:7e:4f:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.160/24 brd 192.168.1.255 scope global qg-91e2a788-b2
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe7e:4f01/64 scope link 
       valid_lft forever preferred_lft forever

root@network01:~# ip netns exec qrouter-88baa567-97f2-48be-883c-a7ae389a60c5 ip r
default via 192.168.1.1 dev qg-91e2a788-b2 
10.10.10.0/24 dev qr-3fa6c9e7-5f  proto kernel  scope link  src 10.10.10.1 
192.168.1.0/24 dev qg-91e2a788-b2  proto kernel  scope link  src 192.168.1.160 

root@network01:~# ovs-vsctl show
babdd49c-701d-456d-bd2c-68b3b0b7df02
    Bridge br-tun
        fail_mode: secure
        Port "gre-c0a83c88"
            Interface "gre-c0a83c88"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="192.168.60.137", out_key=flow, remote_ip="192.168.60.136"}
        Port "gre-c0a83c8c"
            Interface "gre-c0a83c8c"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="192.168.60.137", out_key=flow, remote_ip="192.168.60.140"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qg-91e2a788-b2"
            tag: 2
            Interface "qg-91e2a788-b2"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "qr-3fa6c9e7-5f"
            tag: 1
            Interface "qr-3fa6c9e7-5f"
                type: internal
        Port "tap1e862555-19"
            tag: 1
            Interface "tap1e862555-19"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
    Bridge br-ex
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.3.2"

root@network01:~# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=372.222s, table=0, n_packets=139, n_bytes=20286, idle_age=8, priority=1 actions=NORMAL
 cookie=0x0, duration=363.920s, table=0, n_packets=2, n_bytes=140, idle_age=358, priority=4,in_port=3,dl_vlan=2 actions=strip_vlan,NORMAL
 cookie=0x0, duration=370.564s, table=0, n_packets=21, n_bytes=1750, idle_age=362, priority=2,in_port=3 actions=drop

root@network01:~# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=393.071s, table=0, n_packets=31, n_bytes=2586, idle_age=378, priority=1 actions=NORMAL
 cookie=0x0, duration=390.888s, table=0, n_packets=0, n_bytes=0, idle_age=390, priority=2,in_port=4 actions=drop
 cookie=0x0, duration=383.899s, table=0, n_packets=147, n_bytes=22132, idle_age=10, priority=3,in_port=4,vlan_tci=0x0000 actions=mod_vlan_vid:2,NORMAL
 cookie=0x0, duration=392.879s, table=23, n_packets=0, n_bytes=0, idle_age=392, priority=0 actions=drop

root@network01:~# iptables-save
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:27 2015
*filter
:INPUT ACCEPT [1025:87145]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1054:154428]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
-A INPUT -j neutron-openvswi-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A neutron-openvswi-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP
COMMIT
# Completed on Tue Nov 17 01:21:27 2015
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:27 2015
*mangle
:PREROUTING ACCEPT [1025:87145]
:INPUT ACCEPT [1025:87145]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1054:154428]
:POSTROUTING ACCEPT [1054:154428]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-POSTROUTING - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
:neutron-openvswi-mark - [0:0]
-A PREROUTING -j neutron-openvswi-PREROUTING
-A INPUT -j neutron-openvswi-INPUT
-A FORWARD -j neutron-openvswi-FORWARD
-A OUTPUT -j neutron-openvswi-OUTPUT
-A POSTROUTING -j neutron-openvswi-POSTROUTING
-A neutron-openvswi-PREROUTING -j neutron-openvswi-mark
COMMIT
# Completed on Tue Nov 17 01:21:27 2015
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:27 2015
*nat
:PREROUTING ACCEPT [2:7921]
:INPUT ACCEPT [2:7921]
:OUTPUT ACCEPT [21:1355]
:POSTROUTING ACCEPT [21:1355]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-POSTROUTING - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
:neutron-openvswi-float-snat - [0:0]
:neutron-openvswi-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-openvswi-PREROUTING
-A OUTPUT -j neutron-openvswi-OUTPUT
-A POSTROUTING -j neutron-openvswi-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-openvswi-snat -j neutron-openvswi-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-openvswi-snat
COMMIT
# Completed on Tue Nov 17 01:21:27 2015
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:27 2015
*raw
:PREROUTING ACCEPT [1025:87145]
:OUTPUT ACCEPT [1054:154428]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
-A PREROUTING -j neutron-openvswi-PREROUTING
-A OUTPUT -j neutron-openvswi-OUTPUT
COMMIT
# Completed on Tue Nov 17 01:21:27 2015

root@network01:~# ip netns exec qrouter-88baa567-97f2-48be-883c-a7ae389a60c5 iptables-save
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:33 2015
*filter
:INPUT ACCEPT [60:3680]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-INPUT -m mark --mark 0x1 -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
COMMIT
# Completed on Tue Nov 17 01:21:33 2015
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:33 2015
*mangle
:PREROUTING ACCEPT [108:17351]
:INPUT ACCEPT [60:3680]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-mark - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-mark
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff
-A neutron-l3-agent-mark -i qg-91e2a788-b2 -j MARK --set-xmark 0x2/0xffffffff
COMMIT
# Completed on Tue Nov 17 01:21:33 2015
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:33 2015
*nat
:PREROUTING ACCEPT [56:14677]
:INPUT ACCEPT [8:1006]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i qg-91e2a788-b2 ! -o qg-91e2a788-b2 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-91e2a788-b2 -j SNAT --to-source 192.168.1.160
-A neutron-l3-agent-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 192.168.1.160
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed on Tue Nov 17 01:21:33 2015
# Generated by iptables-save v1.4.21 on Tue Nov 17 01:21:33 2015
*raw
:PREROUTING ACCEPT [108:17351]
:OUTPUT ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
# Completed on Tue Nov 17 01:21:33 2015

root@network01:~# ip netns exec qrouter-88baa567-97f2-48be-883c-a7ae389a60c5 arp -an
? (10.10.10.3) at fa:16:3e:10:31:a8 [ether] on qr-3fa6c9e7-5f
? (192.168.1.1) at 00:0a:cd:1a:2d:6f [ether] on qg-91e2a788-b2

While doing a ping 192.168.1.1 from within the instance, this is what I get on the tcpdump:

root@network01:~# ip netns exec qrouter-88baa567-97f2-48be-883c-a7ae389a60c5 tcpdump -i qg-91e2a788-b2 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qg-91e2a788-b2, link-type EN10MB (Ethernet), capture size 65535 bytes
^C01:25:57.601460 ARP, Request who-has 192.168.1.130 tell 192.168.1.1, length 46
01:25:58.329072 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 26, length 64
01:25:58.601890 ARP, Request who-has 192.168.1.130 tell 192.168.1.1, length 46
01:25:59.331539 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 27, length 64
01:25:59.601446 ARP, Request who-has 192.168.1.130 tell 192.168.1.1, length 46
01:26:00.334374 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 28, length 64
01:26:00.601746 ARP, Request who-has 192.168.1.130 tell 192.168.1.1, length 46
01:26:01.336774 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 29, length 64
01:26:02.338532 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 30, length 64
01:26:03.340653 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 31, length 64
01:26:04.340996 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 32, length 64
01:26:05.342303 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 33, length 64
01:26:06.343264 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 34, length 64
01:26:06.346903 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
01:26:07.344784 IP 192.168.1.160 > 192.168.1.1: ICMP echo request, id 28929, seq 35, length 64
01:26:07.346917 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28

16 packets captured
16 packets received by filter
0 packets dropped by kernel

root@network01:~# tcpdump -i eth1 -n
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
01:26:18.358749 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
01:26:19.354944 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
01:26:20.354942 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
01:26:21.371732 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
01:26:22.370939 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
01:26:23.370962 ARP, Request who-has 192.168.1.1 tell 192.168.1.160, length 28
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

As expected, the tcpdump from outside of the namespace doesn't show anything, but the one from within the qrouter namespace shows ICMP request, but not ICMP reply.

I have looked at https://www.hastexo.com/system/files/neutron_packet_flows-notes-handout.pdf and everything seems to be configured as it should. I have also looked at the various threads on this topic and can't find an answer anywhere.

I have to admit that I am running out of ideas. Please let me know if you need any more information.

Thanks a lot in advance, Bertrand.