Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

1) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

1) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated! I spend far too much time solving this problem Cheers,

jazzsir

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

1) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated! appreciated!.

I spend far too much time solving this problem problem.

Cheers,

jazzsirSean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

1) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not not ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

1) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

1) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in the bottom of "the detail info", please click (more) button to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

1) 2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

2) 3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]# 
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network even though ARP tables of both sides are no problem

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# 
[root@network-node ~]#
ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network even though ARP tables of both sides are no problemnetwork.

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can not ping external network.network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach external network

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can't reach Can not ping from external networknetwork to network node

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
+-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can not ping from external network to network node

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable
Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
 +-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend far too much time solving this problem.

Cheers,

Sean.

Can not ping from external network to network node

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I spend am spending far too much time solving this problem.

Cheers,

Sean.

Can not Cannot ping from external network to network nodefloating IP of tenant router GW

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router GWgateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

This is the

The detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • There are some errors(ConnectionError, HeartbeatTimeout) in log files. - you can check it at the bottom of writing please click (more) button at the bottom of this writing to show all info
  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info", please click (more) button at the bottom of this writing to show all info

The detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • The tap and qr ports are always DOWN (on network-node)
  • There are some errors(ConnectionError, HeartbeatTimeout) in log files. - you can check it at the bottom of writing please click (more) button at the bottom of this writing to show all info
  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info",

The detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#
[root@network-node ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1008.666s, table=0, n_packets=12, n_bytes=936, idle_age=995, priority=1 actions=NORMAL
 cookie=0x0, duration=1008.603s, table=22, n_packets=0, n_bytes=0, idle_age=1008, priority=0 actions=drop
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1046.648s, table=0, n_packets=18, n_bytes=1404, idle_age=1019, priority=0 actions=NORMAL
[root@network-node ~]# 
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:000032a45315fc4f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:42:a9:9e:08:b9:74
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(qr-98c762ea-d7): addr:7f:01:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tapf5eec840-e6): addr:7e:01:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:32:a4:53:15:fc:4f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • The tap and qr ports are always DOWN (on network-node)network-node) - you can check it in "the detail info"
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in "the detail info"
  • There are some errors(ConnectionError, HeartbeatTimeout) in log files. - you can check it at the bottom of writing please click (more) button at the bottom of this writing to show all info
  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response
  • The port of the router_gateway is DOWN (on controller-node) - you can check it at the bottom of "the detail info",

The detail info of my environment:

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#
[root@network-node ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1008.666s, table=0, n_packets=12, n_bytes=936, idle_age=995, priority=1 actions=NORMAL
 cookie=0x0, duration=1008.603s, table=22, n_packets=0, n_bytes=0, idle_age=1008, priority=0 actions=drop
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1046.648s, table=0, n_packets=18, n_bytes=1404, idle_age=1019, priority=0 actions=NORMAL
[root@network-node ~]# 
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:000032a45315fc4f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:42:a9:9e:08:b9:74
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(qr-98c762ea-d7): addr:7f:01:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tapf5eec840-e6): addr:7e:01:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:32:a4:53:15:fc:4f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • The tap and qr ports are always DOWN (on network-node) - you can check it in "the detail info"
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in "the detail info"
  • There are some errors(ConnectionError, HeartbeatTimeout) in log files. - you can check it at the bottom of writing please click (more) button at the bottom of this writing to show all info
  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response

The detail info of my environment:

image description

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#
[root@network-node ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1008.666s, table=0, n_packets=12, n_bytes=936, idle_age=995, priority=1 actions=NORMAL
 cookie=0x0, duration=1008.603s, table=22, n_packets=0, n_bytes=0, idle_age=1008, priority=0 actions=drop
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1046.648s, table=0, n_packets=18, n_bytes=1404, idle_age=1019, priority=0 actions=NORMAL
[root@network-node ~]# 
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:000032a45315fc4f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:42:a9:9e:08:b9:74
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(qr-98c762ea-d7): addr:7f:01:00:00:00:00
     config:    PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tapf5eec840-e6): addr:7e:01:00:00:00:00
     config:    PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:32:a4:53:15:fc:4f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • The tap and qr ports are always DOWN (on network-node) - you can check it in "the detail info"
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in "the detail info"
  • There are some errors(ConnectionError, HeartbeatTimeout) in log files. - you can check it at the bottom of writing please click (more) button at the bottom of this writing to show all info
  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response

The detail info of my environment:

image descriptionimage description

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#
[root@network-node ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1008.666s, table=0, n_packets=12, n_bytes=936, idle_age=995, priority=1 actions=NORMAL
 cookie=0x0, duration=1008.603s, table=22, n_packets=0, n_bytes=0, idle_age=1008, priority=0 actions=drop
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1046.648s, table=0, n_packets=18, n_bytes=1404, idle_age=1019, priority=0 actions=NORMAL
[root@network-node ~]# 
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:000032a45315fc4f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:42:a9:9e:08:b9:74
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(qr-98c762ea-d7): addr:7f:01:00:00:00:00
     config:    PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tapf5eec840-e6): addr:7e:01:00:00:00:00
     config:    PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:32:a4:53:15:fc:4f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.

Cannot ping from external network to floating IP of tenant router gateway

I have followed the Icehouse doc "OpenStack Installation Guide(three-node)". My environment is same as the doc except IP and nodes that are builded as VM using libvirt in CentOS 6.5

http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html

In accordance with the doc.

  • Neutron server is running on controller-node
  • NIC(eth1) for external network on network-node is configured without IP (PROMISC="yes" because the node is VM)

The problem

  • On network node, I can ping to external network but can not ping the floating IP from external network in this stage -> http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron_initial-networks-verify.html

The strange things

  • The tap and qr ports are always DOWN (on network-node) - you can check it in "the detail info"
  • The port of the router_gateway is DOWN (on controller-node) - you can check it in "the detail info"
  • There are some errors(ConnectionError, HeartbeatTimeout) in log files. - you can check it at the bottom of writing please click (more) button at the bottom of this writing to show all info
  • I assigned external IP to br-ex on network-node for test and pinged, then I can capture incoming packets by using "tcpdump -i eth1" on network-node, but there are no 'IP' packets, only 'ARP' packet in captured packets and source host(external node) receives no response

The detail info of my environment:

image descriptionimage description

1) IP addresses and CIDR

-controller-node : 10.0.0.11 (management network)
-network-node : 10.0.0.21 (management network)
                10.0.1.21 (data network)
-compute-node : 10.0.0.31 (management network)
                10.0.1.31 (data network)
-external network gateway : 192.168.125.254
-the floating IP of tenant router gateway : 192.168.125.150
-external network CIDR : 192.168.125.0/24
-demo network CIDR : 172.30.1.0/24

2) network node

[root@network-node ~]# ovs-vsctl show
23804a8f-7c89-4422-9b9f-67bf26a34c51
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-98c762ea-d7"
            tag: 1
            Interface "qr-98c762ea-d7"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf5eec840-e6"
            tag: 1
            Interface "tapf5eec840-e6"
                type: internal
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-9810105a-ed"
            Interface "qg-9810105a-ed"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a00011f"
            Interface "gre-0a00011f"
                type: gre
                options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.11.0"
[root@network-node ~]#
[root@network-node ~]# ip a
.
.
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0
    inet6 fe80::5054:ff:fef6:3107/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fed3:92e2/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2
    inet6 fe80::5054:ff:fe48:c865/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0f1:bff:fe6b:344f/64 scope link 
       valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link 
       valid_lft forever preferred_lft forever
12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 10.0.0.31
13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN 
    link/gre 0.0.0.0 peer 10.0.0.31
    inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0
24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc0e:8cff:fe67:d352/64 scope link 
       valid_lft forever preferred_lft forever
[root@network-node ~]# 
[root@network-node ~]# ip netns list
qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd
qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ip a
7: qg-9810105a-ed: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:34:fd:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.150/24 brd 192.168.125.255 scope global qg-9810105a-ed
    inet6 fe80::f816:3eff:fe34:fdcb/64 scope link 
       valid_lft forever preferred_lft forever
8: qr-98c762ea-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:cb:0c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.30.1.1/24 brd 172.30.1.255 scope global qr-98c762ea-d7
    inet6 fe80::f816:3eff:fecb:c11/64 scope link 
       valid_lft forever preferred_lft forever
19: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: gre0: <NOARP> mtu 1476 qdisc noop state DOWN 
    link/gre 0.0.0.0 brd 0.0.0.0
21: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[root@network-node ~]# 
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.30.1.0      0.0.0.0         255.255.255.0   U     0      0        0 qr-98c762ea-d7
192.168.125.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-9810105a-ed
0.0.0.0         192.168.125.254 0.0.0.0         UG    0      0        0 qg-9810105a-ed
[root@network-node ~]#
[root@network-node ~]# ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
64 bytes from 192.168.125.254: icmp_seq=1 ttl=254 time=0.894 ms
64 bytes from 192.168.125.254: icmp_seq=2 ttl=254 time=0.917 ms
[root@network-node ~]# ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
From 192.168.125.58 icmp_seq=2 Destination Host Unreachable
From 192.168.125.58 icmp_seq=3 Destination Host Unreachable
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.150
PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data.
64 bytes from 192.168.125.150: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.125.150: icmp_seq=2 ttl=64 time=0.107 ms
[root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 ping 192.168.125.254
PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data.
From 192.168.125.150 icmp_seq=2 Destination Host Unreachable
From 192.168.125.150 icmp_seq=3 Destination Host Unreachable
From 192.168.125.150 icmp_seq=4 Destination Host Unreachable 
  [root@network-node ~]#
  [root@network-node ~]# nova secgroup-list
+--------------------------------------+---------+-------------+
| Id                                   | Name    | Description |
+--------------------------------------+---------+-------------+
| 550906cc-40db-4e48-8fa1-981e75009d51 | default | default     |
+--------------------------------------+---------+-------------+
[root@network-node ~]# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
|             |           |         |           | default      |
+-------------+-----------+---------+-----------+--------------+
[root@network-node ~]#
[root@network-node ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1008.666s, table=0, n_packets=12, n_bytes=936, idle_age=995, priority=1 actions=NORMAL
 cookie=0x0, duration=1008.603s, table=22, n_packets=0, n_bytes=0, idle_age=1008, priority=0 actions=drop
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1046.648s, table=0, n_packets=18, n_bytes=1404, idle_age=1019, priority=0 actions=NORMAL
[root@network-node ~]# 
[root@network-node ~]# 
[root@network-node ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:000032a45315fc4f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:42:a9:9e:08:b9:74
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(qr-98c762ea-d7): addr:7f:01:00:00:00:00
     config:    PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tapf5eec840-e6): addr:7e:01:00:00:00:00
     config:    PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:32:a4:53:15:fc:4f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

3) controller-node

[root@controller-node]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24    |
| e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net  | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 |
+--------------------------------------+----------+-------------------------------------------------------+
[root@controller-node]# neutron subnet-list
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| id                                   | name        | cidr             | allocation_pools                                       |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
| a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet  | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} |
| c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24    | {"start": "172.30.1.2", "end": "172.30.1.254"}         |
+--------------------------------------+-------------+------------------+--------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron port-list
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 9810105a-edf5-41bc-a140-81ccf71f6bc4 |      | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| 98c762ea-d7f7-4c1d-9b74-73efc9990236 |      | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"}      |
| f5eec840-e629-448b-ba9a-fbcd60501247 |      | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"}      |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# neutron router-show demo-router
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} |
| id                    | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | c94f1dc5870a4d06a8b6ba947e1ac554                                            |
+-----------------------+-----------------------------------------------------------------------------+
[root@controller-node]# 
[root@controller-node]# 
[root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4
+-----------------------+----------------------------------------------------------------------------------------+
| Field                 | Value                                                                                  |
+-----------------------+----------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                   |
| allowed_address_pairs |                                                                                        |
| binding:host_id       | os-network                                                                             |
| binding:profile       | {}                                                                                     |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                         |
| binding:vif_type      | ovs                                                                                    |
| binding:vnic_type     | normal                                                                                 |
| device_id             | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555                                                   |
| device_owner          | network:router_gateway                                                                 |
| extra_dhcp_opts       |                                                                                        |
| fixed_ips             | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} |
| id                    | 9810105a-edf5-41bc-a140-81ccf71f6bc4                                                   |
| mac_address           | fa:16:3e:34:fd:cb                                                                      |
| name                  |                                                                                        |
| network_id            | e5f7b93c-475c-4c9d-95e4-8d1cf7728013                                                   |
| security_groups       |                                                                                        |
| status                | DOWN                                                                                   |
| tenant_id             |                                                                                        |
    +-----------------------+----------------------------------------------------------------------------------------+

Errors in log files.

1) l3-agent.log on network-node

-Error 1
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to consume message from queue: connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1929 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1929 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

-Error 2
1929 ERROR neutron.openstack.common.rpc.impl_qpid [req-600f959c-433b-4d7c-9199-6907b3c047b1 None] Failed to publish message to topic 'q-l3-plugin': connection aborted
1929 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1929 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 633, in _publisher_send
1929 TRACE neutron.openstack.common.rpc.impl_qpid     publisher = cls(self.conf, self.session, topic)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 397, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     super(TopicPublisher, self).__init__(conf, session, node_name)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 327, in __init__
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.reconnect(session)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 331, in reconnect
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.sender = session.sender(self.address)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 597, in sender
1929 TRACE neutron.openstack.common.rpc.impl_qpid     sender._ewait(lambda: sender.linked)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 804, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.session._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1929 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1929 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1929 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1929 TRACE neutron.openstack.common.rpc.impl_qpid ConnectionError: connection aborted

2) openvswitch-agent.log on network-node

1993 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
1993 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
1993 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
1993 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
1993 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
1993 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
1993 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
1993 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
1993 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

3) server.log on controller-node

1829 ERROR neutron.api.v2.resource [req-25251071-07d4-460e-b15a-77a126b46f22 None] update failed
1829 TRACE neutron.api.v2.resource Traceback (most recent call last):
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
1829 TRACE neutron.api.v2.resource     result = method(request=request, **args)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 478, in update
1829 TRACE neutron.api.v2.resource     allow_bulk=self._allow_bulk)
1829 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 597, in prepare_request_body
1829 TRACE neutron.api.v2.resource     raise webob.exc.HTTPBadRequest(msg)
1829 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute status

Any hints would be much appreciated!.

I am spending far too much time solving this problem.

Cheers,

Sean.