Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Connection lost between OVS patch ports?

I'm setting up Kilo KVM region and got a problem on network configuration. I have been stuck in there for a couple of days now, so any suggestions are much appreciated!

This is three a nodes environment:

kilocontroller (management network IP 192.168.155.188 on eth0)

kilonetwork (management and external network IP 192.168.155.189 on br-ex, Data network IP 10.10.1.20 on eth1)

kilocompute (management network IP 192.168.155.190 on eth0, Data network IP 10.10.1.21 on eth1)

-All notes are VMs running RHEL v7.1 on vCenter v5.5.

-I’m following the KILO RHEL installation guide (http://docs.openstack.org/kilo/install-guide/install/yum/content/). So both tenant and tunnel use GRE.

I'm able to provision VM successfully but VM can't get IP address, by tracking the DHCP request I found the communication lost between OVS patch ports in compute node.

For example in case of VM VM-20151104-2 , the DHCP communication chain on compute node should be:

VM-20151104-2 -> tapb2517833-13 (qbrb2517833-13) qvbb2517833-13 -> qvob2517833-13 (br-int) patch-tun -> patch-int (br-tun) gre-0a0a0114 -> eth1 ====> Network Node ......

While I saw the DHCP request on port tapb2517833-13, qvbb2517833-13, qvob2517833-13, patch-tun but it didn't reach patch-int. At the same time I saw request on eth1 on both kilocompute and kilonetwork nodes (it never goes further on network node).

So the questions are:

1) Why DHCP lost between OVS patch ports? (Yes OVS peer is there)

2) How come eth1 on kilocompute get the traffic if OVS patch connection is broken?

Here is what I have on kilocompute:

[root@kilocompute ~]# brctl show
bridge name bridge id       STP enabled interfaces
qbrb2517833-13      8000.ce00be78fda3   no      qvbb2517833-13
                            tapb2517833-13

[root@kilocompute ~]# ovs-vsctl show
ff6f0cf5-f7a9-4572-89cd-4969f66e7516
    Bridge br-tun
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0a0a0114"
            Interface "gre-0a0a0114"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="10.10.1.21", out_key=flow, remote_ip="10.10.1.20"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvob2517833-13"
            tag: 1
            Interface "qvob2517833-13"
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.3.1"
[root@kilocompute ~]# 


[root@kilocompute ~]# ovs-ofctl dump-ports br-int
OFPST_PORT reply (xid=0x2): 3 ports
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=20, bytes=4656, drop=0, errs=0, coll=0
  port  2: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=20, bytes=4656, drop=0, errs=0, coll=0
  port  3: rx pkts=20, bytes=4656, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
[root@kilocompute ~]#


[root@kilocompute ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=40163.854s, table=0, n_packets=20, n_bytes=4656, idle_age=37601, priority=1 actions=NORMAL
 cookie=0x0, duration=40163.770s, table=23, n_packets=0, n_bytes=0, idle_age=40163, priority=0 actions=drop
[root@kilocompute ~]# 


[root@kilocompute ~]# ovs-ofctl dump-ports br-tun
OFPST_PORT reply (xid=0x2): 3 ports
  port  4: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=20, bytes=4656, drop=0, errs=0, coll=0
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
  port  3: rx pkts=20, bytes=4656, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
[root@kilocompute ~]# 



[root@kilocompute ~]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=40103.205s, table=0, n_packets=0, n_bytes=0, idle_age=40103, priority=0 actions=drop
 cookie=0x0, duration=40103.324s, table=0, n_packets=20, n_bytes=4656, idle_age=37542, priority=1,in_port=3 actions=resubmit(,2)
 cookie=0x0, duration=40101.888s, table=0, n_packets=0, n_bytes=0, idle_age=40101, priority=1,in_port=4 actions=resubmit(,3)
 cookie=0x0, duration=40103.121s, table=2, n_packets=0, n_bytes=0, idle_age=40103, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=40103.039s, table=2, n_packets=20, n_bytes=4656, idle_age=37542, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=40102.952s, table=3, n_packets=0, n_bytes=0, idle_age=40102, priority=0 actions=drop
 cookie=0x0, duration=40100.480s, table=3, n_packets=0, n_bytes=0, idle_age=40100, priority=1,tun_id=0x64 actions=mod_vlan_vid:1,resubmit(,10)
 cookie=0x0, duration=40102.865s, table=4, n_packets=0, n_bytes=0, idle_age=40102, priority=0 actions=drop
 cookie=0x0, duration=40102.782s, table=10, n_packets=0, n_bytes=0, idle_age=40102, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:3
 cookie=0x0, duration=40102.700s, table=20, n_packets=0, n_bytes=0, idle_age=40102, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=40102.579s, table=22, n_packets=0, n_bytes=0, idle_age=40102, priority=0 actions=drop
 cookie=0x0, duration=40100.560s, table=22, n_packets=20, n_bytes=4656, idle_age=37542, dl_vlan=1 actions=strip_vlan,set_tunnel:0x64,output:4
[root@kilocompute ~]#

This is what I have while tracing DHCP on kilocompute:

[root@kilocompute ~]# ip link add name snooper0 type dummy

[root@kilocompute ~]# ip link set dev snooper0 up

[root@kilocompute ~]# ovs-vsctl add-port br-tun snooper0

[root@kilocompute ~]# ovs-vsctl -- set Bridge br-tun mirrors=@m  -- --id=@snooper0 get Port snooper0  -- --id=@patch-int get Port patch-int -- --id=@m create Mirror name=mymirror select-dst-port=@patch-int select-src-port=@patch-int output-port=@snooper0 select_all=1
c02fbd1a-72d2-4b9d-9ff7-0dc78966f189

[root@kilocompute ~]# tcpdump -i snooper0 -vv
tcpdump: WARNING: snooper0: no IPv4 address assigned
tcpdump: listening on snooper0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
**<========= Note: nothing here**

**There is traffic on patch-tun=======>**

[root@kilocompute ~]# ip link add name snooper1 type dummy

[root@kilocompute ~]# ip link set dev snooper1 up

[root@kilocompute ~]# ovs-vsctl add-port br-int snooper1

[root@kilocompute ~]# ovs-vsctl -- set Bridge br-int mirrors=@m  -- --id=@snooper1 get Port snooper1  -- --id=@patch-tun get Port patch-tun -- --id=@m create Mirror name=mymirror1 select-dst-port=@patch-tun select-src-port=@patch-tun output-port=@snooper1 select_all=1
2de72178-00b8-4ca1-b29c-2aaf1dddc250

[root@kilocompute ~]# tcpdump -i snooper1 -vv
tcpdump: WARNING: snooper1: no IPv4 address assigned
tcpdump: listening on snooper1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:02:01.619038 IP6 (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff6a:2a4e to_ex { }]
11:02:01.622559 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 318)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:6a:2a:4e (oui Unknown), length 290, xid 0x81a85832, Flags [none] (0x0000)
      Client-Ethernet-Address fa:16:3e:6a:2a:4e (oui Unknown)
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Discover
        Client-ID Option 61, length 7: ether fa:16:3e:6a:2a:4e
        MSZ Option 57, length 2: 576
        Parameter-Request Option 55, length 9: 
          Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
          Domain-Name, MTU, BR, NTP
          Classless-Static-Route
        Vendor-Class Option 60, length 12: "udhcp 1.20.1"
        Hostname Option 12, length 6: "cirros"
……

**There is traffic on eth1=======>**

[root@kilocompute ~]# tcpdump -i eth1 -vv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
10:56:31.628805 IP (tos 0x0, ttl 64, id 46917, offset 0, flags [DF], proto GRE (47), length 360)
    10.10.1.21 > 10.10.1.20: GREv0, Flags [key present], key=0x64, length 340
    IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 318)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:6a:2a:4e (oui Unknown), length 290, xid 0x4c3f0e04, secs 60, Flags [none] (0x0000)
      Client-Ethernet-Address fa:16:3e:6a:2a:4e (oui Unknown)
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Discover
        Client-ID Option 61, length 7: ether fa:16:3e:6a:2a:4e
        MSZ Option 57, length 2: 576
        Parameter-Request Option 55, length 9: 
          Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
          Domain-Name, MTU, BR, NTP
          Classless-Static-Route
        Vendor-Class Option 60, length 12: "udhcp 1.20.1"
        Hostname Option 12, length 6: "cirros"
10:56:31.629206 IP (tos 0xc0, ttl 64, id 40905, offset 0, flags [none], proto ICMP (1), length 388)
    10.10.1.20 > 10.10.1.21: ICMP host 10.10.1.20 unreachable - admin prohibited, length 368
    IP (tos 0x0, ttl 64, id 46917, offset 0, flags [DF], proto GRE (47), length 360)
    10.10.1.21 > 10.10.1.20: GREv0, Flags [key present], key=0x64, length 340
    IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 318)
……

[root@kilocompute ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:be:1e:28 brd ff:ff:ff:ff:ff:ff
    inet 192.168.155.190/24 brd 192.168.155.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:be:65:12 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.21/24 brd 10.10.1.255 scope global eth1
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether e2:2d:9e:66:d0:0d brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 6a:04:64:ed:75:4c brd ff:ff:ff:ff:ff:ff
6: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 96:85:8b:6d:44:43 brd ff:ff:ff:ff:ff:ff
7: qbrb2517833-13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether ce:00:be:78:fd:a3 brd ff:ff:ff:ff:ff:ff
8: qvob2517833-13: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 6a:f3:f4:05:61:4a brd ff:ff:ff:ff:ff:ff
9: qvbb2517833-13: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrb2517833-13 state UP qlen 1000
    link/ether ce:00:be:78:fd:a3 brd ff:ff:ff:ff:ff:ff
16: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN 
    link/ether 5a:56:1e:68:05:2f brd ff:ff:ff:ff:ff:ff
17: snooper0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UNKNOWN 
    link/ether 02:64:d2:1a:09:3c brd ff:ff:ff:ff:ff:ff
19: snooper1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UNKNOWN 
    link/ether 1e:04:7c:1a:71:be brd ff:ff:ff:ff:ff:ff
20: tapb2517833-13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrb2517833-13 state UNKNOWN qlen 500
    link/ether fe:16:3e:6a:2a:4e brd ff:ff:ff:ff:ff:ff

On kilonetwork:

[root@kilonetwork ~]# ip netns list
qdhcp-5c401694-983b-4194-8a34-e8b6332e2e9f

[root@kilonetwork ~]# ip netns exec qdhcp-5c401694-983b-4194-8a34-e8b6332e2e9f ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
9: tap33a1fdcd-cf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:0c:07:59 brd ff:ff:ff:ff:ff:ff
    inet 10.10.100.100/24 brd 10.10.100.255 scope global tap33a1fdcd-cf
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe0c:759/64 scope link 
       valid_lft forever preferred_lft forever

[root@kilonetwork ~]#  ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000626b76054445
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-int): addr:66:69:35:fb:2a:cf
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(gre-0a0a0115): addr:9a:bf:a2:a8:72:86
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:62:6b:76:05:44:45
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root@kilonetwork ~]#

[root@kilonetwork ~]# ovs-ofctl dump-ports br-tun
OFPST_PORT reply (xid=0x2): 3 ports
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
  port  1: rx pkts=8, bytes=648, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
  port  2: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=3, bytes=230, drop=0, errs=0, coll=0
[root@kilonetwork ~]# 

[root@kilonetwork ~]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=59040.495s, table=0, n_packets=0, n_bytes=0, idle_age=59040, priority=0 actions=drop
 cookie=0x0, duration=59040.586s, table=0, n_packets=8, n_bytes=648, idle_age=41842, priority=1,in_port=1 actions=resubmit(,2)
 cookie=0x0, duration=40306.188s, table=0, n_packets=0, n_bytes=0, idle_age=42746, priority=1,in_port=2 actions=resubmit(,3)
 cookie=0x0, duration=59040.401s, table=2, n_packets=0, n_bytes=0, idle_age=59040, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=59040.312s, table=2, n_packets=8, n_bytes=648, idle_age=41842, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=59040.221s, table=3, n_packets=0, n_bytes=0, idle_age=59040, priority=0 actions=drop
 cookie=0x0, duration=41850.317s, table=3, n_packets=0, n_bytes=0, idle_age=41850, priority=1,tun_id=0x64 actions=mod_vlan_vid:1,resubmit(,10)
 cookie=0x0, duration=59040.124s, table=4, n_packets=0, n_bytes=0, idle_age=59040, priority=0 actions=drop
 cookie=0x0, duration=59040.031s, table=10, n_packets=0, n_bytes=0, idle_age=59040, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0x0, duration=59039.897s, table=20, n_packets=0, n_bytes=0, idle_age=59039, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=59039.799s, table=22, n_packets=5, n_bytes=418, idle_age=41850, priority=0 actions=drop
 cookie=0x0, duration=41850.409s, table=22, n_packets=3, n_bytes=230, idle_age=41842, hard_age=40306, dl_vlan=1 actions=strip_vlan,set_tunnel:0x64,output:2
[root@kilonetwork ~]# 


[root@kilonetwork ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:00004aaed8030e4c
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(int-br-ex): addr:12:5a:88:69:97:45
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(patch-tun): addr:86:49:dc:a3:d1:22
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(tap33a1fdcd-cf): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:4a:ae:d8:03:0e:4c
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root@kilonetwork ~]# 

[root@kilonetwork ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=60742.863s, table=0, n_packets=8, n_bytes=648, idle_age=43542, priority=1 actions=NORMAL
 cookie=0x0, duration=60741.811s, table=0, n_packets=1178, n_bytes=95410, idle_age=60369, priority=2,in_port=1 actions=drop
 cookie=0x0, duration=60742.772s, table=23, n_packets=0, n_bytes=0, idle_age=60742, priority=0 actions=drop
[root@kilonetwork ~]#

I think I might be missing something quite obvious but can't work it out, I'm happy to provide any other information you need. Any thoughts are much appreciated!