Revision history [back]

click to hide/show revision 1
initial version

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
8d98dca1-d77c-4eb9-81c8-19e093ae687f
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens224"
            Interface "ens224"
        Port "qg-3dc8a875-c9"
            Interface "qg-3dc8a875-c9"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap9e25d28b-e2"
            tag: 2
            Interface "tap9e25d28b-e2"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-dfb09db5-40"
            tag: 2
            Interface "qr-dfb09db5-40"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.3.1"

    [root@compute neutron]# ovs-vsctl show
13a9168a-dcda-446b-b286-3a69dfe41cca
    Bridge br-int
        fail_mode: secure
        Port "qvodefff050-2d"
            tag: 5
            Interface "qvodefff050-2d"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvoac304f4b-01"
            tag: 5
            Interface "qvoac304f4b-01"
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I believe there is something blocking the GRE connectivity?

Any help here would be appreciated.

Thanks

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
8d98dca1-d77c-4eb9-81c8-19e093ae687f
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens224"
            Interface "ens224"
        Port "qg-3dc8a875-c9"
            Interface "qg-3dc8a875-c9"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap9e25d28b-e2"
            tag: 2
            Interface "tap9e25d28b-e2"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-dfb09db5-40"
            tag: 2
            Interface "qr-dfb09db5-40"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.3.1"

    [root@compute neutron]# ovs-vsctl show
13a9168a-dcda-446b-b286-3a69dfe41cca
    Bridge br-int
        fail_mode: secure
        Port "qvodefff050-2d"
            tag: 5
            Interface "qvodefff050-2d"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvoac304f4b-01"
            tag: 5
            Interface "qvoac304f4b-01"
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I believe there is something blocking the GRE connectivity?

Any help here would be appreciated.

Thanks

The pointer to the document help me understand the traffic flow, but did not resolve the issue.

I see that the traffic from the VM is being sent with tag:1

[root@compute ~]# ovs-vsctl show 13a9168a-dcda-446b-b286-3a69dfe41cca Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvoac304f4b-01" tag: 1 Interface "qvoac304f4b-01"

The GRE tunnel is also setup correctly:

Port "gre-0f000003" Interface "gre-0f000003" type: gre options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"} ovs_version: "2.3.1"

On the Neutron server, using ovs-ofctl dump-flows br-tun, I see the following for ID = 1:

cookie=0x0, duration=739.328s, table=3, n_packets=0, n_bytes=0, idle_age=739, priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10)

This says it redirects to group 10. Group 10 shows the following:

cookie=0x0, duration=714.263s, table=10, n_packets=0, n_bytes=0, idle_age=714, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Is this a problem?

Also, I see the following on the Neutron server:

[root@neutron ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000faa4a7adb34d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 2(int-br-ex): addr:ee:dd:3e:29:6e:d1
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-tun): addr:0e:9e:71:79:b7:31
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(tap9e25d28b-e2): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(qr-dfb09db5-40): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:fa:a4:a7:ad:b3:4d
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Does this mean the link to the qrouter is down?

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
8d98dca1-d77c-4eb9-81c8-19e093ae687f
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens224"
            Interface "ens224"
        Port "qg-3dc8a875-c9"
            Interface "qg-3dc8a875-c9"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap9e25d28b-e2"
            tag: 2
            Interface "tap9e25d28b-e2"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-dfb09db5-40"
            tag: 2
            Interface "qr-dfb09db5-40"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.3.1"

    [root@compute neutron]# ovs-vsctl show
13a9168a-dcda-446b-b286-3a69dfe41cca
    Bridge br-int
        fail_mode: secure
        Port "qvodefff050-2d"
            tag: 5
            Interface "qvodefff050-2d"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvoac304f4b-01"
            tag: 5
            Interface "qvoac304f4b-01"
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I believe there is something blocking the GRE connectivity?

Any help here would be appreciated.

Thanks

The pointer to the document help me understand the traffic flow, but did not resolve the issue.

I see that the traffic from the VM is being sent with tag:1

[root@compute ~]# ovs-vsctl show 13a9168a-dcda-446b-b286-3a69dfe41cca Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvoac304f4b-01" tag: 1 Interface "qvoac304f4b-01"

The GRE tunnel is also setup correctly:

Port "gre-0f000003" Interface "gre-0f000003" type: gre options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"} ovs_version: "2.3.1"

On the Neutron server, using ovs-ofctl dump-flows br-tun, I see the following for ID = 1:

cookie=0x0, duration=739.328s, table=3, n_packets=0, n_bytes=0, idle_age=739, priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10)

This says it redirects to group 10. Group 10 shows the following:

cookie=0x0, duration=714.263s, table=10, n_packets=0, n_bytes=0, idle_age=714, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Is this a problem?

Also, I see the following on the Neutron server:

[root@neutron ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000faa4a7adb34d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 2(int-br-ex): addr:ee:dd:3e:29:6e:d1
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-tun): addr:0e:9e:71:79:b7:31
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(tap9e25d28b-e2): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(qr-dfb09db5-40): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:fa:a4:a7:ad:b3:4d
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Does this mean the link to the qrouter is down?

I did a tcpdump on the ethernet device running the GRE tunnel. It does not appear that any traffic is flowing.

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
8d98dca1-d77c-4eb9-81c8-19e093ae687f
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens224"
            Interface "ens224"
        Port "qg-3dc8a875-c9"
            Interface "qg-3dc8a875-c9"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap9e25d28b-e2"
            tag: 2
            Interface "tap9e25d28b-e2"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-dfb09db5-40"
            tag: 2
            Interface "qr-dfb09db5-40"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.3.1"

    [root@compute neutron]# ovs-vsctl show
13a9168a-dcda-446b-b286-3a69dfe41cca
    Bridge br-int
        fail_mode: secure
        Port "qvodefff050-2d"
            tag: 5
            Interface "qvodefff050-2d"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvoac304f4b-01"
            tag: 5
            Interface "qvoac304f4b-01"
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I believe there is something blocking the GRE connectivity?

Any help here would be appreciated.

Thanks

The pointer to the document help me understand the traffic flow, but did not resolve the issue.

I see that the traffic from the VM is being sent with tag:1

[root@compute ~]# ovs-vsctl show 13a9168a-dcda-446b-b286-3a69dfe41cca Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvoac304f4b-01" tag: 1 Interface "qvoac304f4b-01"

The GRE tunnel is also setup correctly:

Port "gre-0f000003" Interface "gre-0f000003" type: gre options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"} ovs_version: "2.3.1"

On the Neutron server, using ovs-ofctl dump-flows br-tun, I see the following for ID = 1:

cookie=0x0, duration=739.328s, table=3, n_packets=0, n_bytes=0, idle_age=739, priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10)

This says it redirects to group 10. Group 10 shows the following:

cookie=0x0, duration=714.263s, table=10, n_packets=0, n_bytes=0, idle_age=714, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Is this a problem?

Also, I see the following on the Neutron server:

[root@neutron ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000faa4a7adb34d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 2(int-br-ex): addr:ee:dd:3e:29:6e:d1
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-tun): addr:0e:9e:71:79:b7:31
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(tap9e25d28b-e2): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(qr-dfb09db5-40): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:fa:a4:a7:ad:b3:4d
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Does this mean the link to the qrouter is down?

I did a tcpdump on the ethernet device running the GRE tunnel. It does not appear that any traffic is flowing.

Also, I do not see the MAC address of the VM in the br-int device when using ovs-ofctl show br-int

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
8d98dca1-d77c-4eb9-81c8-19e093ae687f
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens224"
            Interface "ens224"
        Port "qg-3dc8a875-c9"
            Interface "qg-3dc8a875-c9"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap9e25d28b-e2"
            tag: 2
            Interface "tap9e25d28b-e2"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-dfb09db5-40"
            tag: 2
            Interface "qr-dfb09db5-40"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.3.1"

    [root@compute neutron]# ovs-vsctl show
13a9168a-dcda-446b-b286-3a69dfe41cca
    Bridge br-int
        fail_mode: secure
        Port "qvodefff050-2d"
            tag: 5
            Interface "qvodefff050-2d"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvoac304f4b-01"
            tag: 5
            Interface "qvoac304f4b-01"
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I believe there is something blocking the GRE connectivity?

Any help here would be appreciated.

Thanks

The pointer to the document help me understand the traffic flow, but did not resolve the issue.

I see that the traffic from the VM is being sent with tag:1

[root@compute ~]# ovs-vsctl show 13a9168a-dcda-446b-b286-3a69dfe41cca Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvoac304f4b-01" tag: 1 Interface "qvoac304f4b-01"

The GRE tunnel is also setup correctly:

Port "gre-0f000003" Interface "gre-0f000003" type: gre options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"} ovs_version: "2.3.1"

On the Neutron server, using ovs-ofctl dump-flows br-tun, I see the following for ID = 1:

cookie=0x0, duration=739.328s, table=3, n_packets=0, n_bytes=0, idle_age=739, priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10)

This says it redirects to group 10. Group 10 shows the following:

cookie=0x0, duration=714.263s, table=10, n_packets=0, n_bytes=0, idle_age=714, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Is this a problem?

Also, I see the following on the Neutron server:

[root@neutron ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000faa4a7adb34d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 2(int-br-ex): addr:ee:dd:3e:29:6e:d1
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-tun): addr:0e:9e:71:79:b7:31
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(tap9e25d28b-e2): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(qr-dfb09db5-40): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:fa:a4:a7:ad:b3:4d
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Does this mean the link to the qrouter is down?

I did a tcpdump on the ethernet device running the GRE tunnel. It I am seeing DCHP traffic from the compute to the neutron server and the neutron server is seeing it.

No response to the DHCP request is being sent back.

I have looked at a troubleshooting article, and it is saying to look at table 2 on the br-tun device.

Table 2 does not appear that have any traffic entry for the ID of my VM.

Also, table 21 which is flowing.

Also, I do not see the MAC address of the VM in the br-int device when using ovs-ofctl show br-intsupposed to be where the DHCP requests go appears to be empty as well.

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
8d98dca1-d77c-4eb9-81c8-19e093ae687f
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens224"
            Interface "ens224"
        Port "qg-3dc8a875-c9"
            Interface "qg-3dc8a875-c9"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap9e25d28b-e2"
            tag: 2
            Interface "tap9e25d28b-e2"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-dfb09db5-40"
            tag: 2
            Interface "qr-dfb09db5-40"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.3.1"

    [root@compute neutron]# ovs-vsctl show
13a9168a-dcda-446b-b286-3a69dfe41cca
    Bridge br-int
        fail_mode: secure
        Port "qvodefff050-2d"
            tag: 5
            Interface "qvodefff050-2d"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvoac304f4b-01"
            tag: 5
            Interface "qvoac304f4b-01"
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I believe used the following article for troubleshooting: http://dischord.org/2015/03/09/troubleshooting-openstack-neutron-networking-part-one/

There is traffic flowing between compute and neutron as I am seeing the DHCP requests on the neutron server via tcpdump.

Looking at the flows on the OVS br-tun, it appears something is wrong there (the segmenation_id is something blocking the GRE connectivity?

Any help here would be appreciated.

Thanks

The pointer to the document help me understand the traffic flow, but did not resolve the issue.

I see that the traffic from the VM is being sent with tag:1

[root@compute ~]# ovs-vsctl show 13a9168a-dcda-446b-b286-3a69dfe41cca Bridge br-int fail_mode: secure Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvoac304f4b-01" tag: 1 Interface "qvoac304f4b-01"

The GRE tunnel is also setup correctly:

Port "gre-0f000003" Interface "gre-0f000003" type: gre options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"} ovs_version: "2.3.1"

On the Neutron server, using ovs-ofctl dump-flows br-tun, I see the following for ID = 1:

cookie=0x0, duration=739.328s, table=3, n_packets=0, n_bytes=0, idle_age=739, priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10)

This says it redirects to group 10. Group 10 shows the following:

cookie=0x0, duration=714.263s, table=10, n_packets=0, n_bytes=0, idle_age=714, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Is this a problem?

Also, I see the following on the Neutron server:1)

[root@neutron ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000faa4a7adb34d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 2(int-br-ex): addr:ee:dd:3e:29:6e:d1
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-tun): addr:0e:9e:71:79:b7:31
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(tap9e25d28b-e2): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(qr-dfb09db5-40): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:fa:a4:a7:ad:b3:4d
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
dump-flows br-tun | grep 0x1
 cookie=0x0, duration=3553.900s, table=3, n_packets=0, n_bytes=0, idle_age=3553, priority=1,tun_id=0x1 actions=mod_vlan_vid:2,resubmit(,10)
 cookie=0x0, duration=3554.071s, table=22, n_packets=2, n_bytes=140, idle_age=3316, dl_vlan=2 actions=strip_vlan,set_tunnel:0x1,output:2

Does this mean the link to the qrouter is down?

I did a tcpdump on the ethernet device running the GRE tunnel. I am seeing DCHP traffic from the compute to the neutron server and the neutron server is seeing it.

No response to the DHCP request is being sent back.

I have looked at a troubleshooting article, and it is saying to look at table 2 on the br-tun device.

Table 2 does not have any entry for the ID of my VM.

Also, table 21 which is supposed to be where the DHCP requests go appears to be empty as well.Please help.

Thanks, Doug

cannot ping tenant network in cirros using neutron networking - Juno

I have setup an Juno OpenStack cloud in my lab. I have 3 machines running on a VMware vCenter cluster:

Box #1 - Controller & Glance Box #2 - Compute Box #3 - Neutron

Box #1 has 1 network connection (management network) Box #2 has 2 network connections (management network & GRE) Box #3 has 3 network connections (management, GRE, & External net)

I am able to start instances without any issue. I can get to them via VNC proxy, but they have no connectivity to the tenant network or the router.

There are no errors or warnings in any of the nova or neutron logs.

The ovs-vsctl show output shows the connection between the 2 boxes on the GRE network is up:

 [root@neutron neutron]# ovs-vsctl show
    Bridge br-tun
        fail_mode: secure
        Port "gre-0f000002"
            Interface "gre-0f000002"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.3", out_key=flow, remote_ip="15.0.0.2"}

    [root@compute neutron]# ovs-vsctl show
        Port "gre-0f000003"
            Interface "gre-0f000003"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="15.0.0.2", out_key=flow, remote_ip="15.0.0.3"}
    ovs_version: "2.3.1"

I tried setting the ip address manually in the Cirros instance and it still won't ping the tenant router.

I used the following article for troubleshooting: http://dischord.org/2015/03/09/troubleshooting-openstack-neutron-networking-part-one/

There is traffic flowing between compute and neutron as I am seeing the DHCP requests on the neutron server via tcpdump.

I did a tcpdump on the tap interface in the br-tun bridge. It's not showing any data flowing at all.

[root@neutron ~]# ip netns list | grep c7c59f68-e276-4899-87ac-1ad17c3e28ad
qdhcp-c7c59f68-e276-4899-87ac-1ad17c3e28ad
[root@neutron ~]# ip netns exec qdhcp-c7c59f68-e276-4899-87ac-1ad17c3e28ad ip li
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
12: tapa4f79f35-6f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
    link/ether fa:16:3e:04:63:15 brd ff:ff:ff:ff:ff:ff
[root@neutron ~]# ovs-vsctl show | grep -A1 tapa4f79f35-6f
        Port "tapa4f79f35-6f"
            tag: 1
            Interface "tapa4f79f35-6f"
                type: internal
[root@neutron ~]# ip netns exec qdhcp-c7c59f68-e276-4899-87ac-1ad17c3e28ad tcpdump -i tapa4f79f35-6f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapa4f79f35-6f, link-type EN10MB (Ethernet), capture size 65535 bytes

Looking at the flows on the OVS br-tun, it appears something is wrong there (the segmenation_id is 1)

[root@neutron ~]# ovs-ofctl dump-flows br-tun | grep 0x1
 cookie=0x0, duration=3553.900s, table=3, n_packets=0, n_bytes=0, idle_age=3553, priority=1,tun_id=0x1 actions=mod_vlan_vid:2,resubmit(,10)
 cookie=0x0, duration=3554.071s, table=22, n_packets=2, n_bytes=140, idle_age=3316, dl_vlan=2 actions=strip_vlan,set_tunnel:0x1,output:2

Please help.

Thanks, Doug