Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

ovs (with --dpdk) doesn't forward packets to vxlan tunnels in an openstack environment

I have a compute host with Debian 8, OVS 2.4.90, compiled for dpdk 2.0.0 and running with dpdk support (--dpdk), in an openstack environment with ODL on the controller as the OVS manager (so no neutron agent). A VM is created on the same compute I mentioned. The VM tries to obtain an IP from dhcp. It sends BOOTP packets.

Those packets are matched against:

cookie=0x0, duration=575.331s, table=110, n_packets=84, n_bytes=16056, priority=16383,reg0=0x1,tun_id=0xc9,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:8,output:4,output:1,output:5,output:6,output:7

This is how the bridge looks like:

OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:000072b0e2b1ed4c
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(vxlan-10.53.6.1): addr:7e:f9:5f:c9:86:6a
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-ext): addr:96:ee:48:d0:58:23
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(vxlan-10.53.6.1): addr:e2:f9:b2:7b:47:9f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(vxlan-10.53.6.1): addr:8e:26:fc:b8:b9:cb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 6(vxlan-10.53.6.1): addr:6a:69:b6:9a:47:56
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 7(vxlan-10.53.6.1): addr:de:ed:fc:3f:7c:c3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 8(tap79cdb725-73): addr:fe:16:3e:b0:bc:10
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:72:b0:e2:b1:ed:4c
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0

The local_ip configured on the bridge:

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-vsctl get Open_vSwitch . other_config:local_ip
"10.53.6.16"

The interface:

root@BASE-CCP-CPN-N0003-NETCLM:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr b0:5a:da:02:10:20
          inet addr:10.53.6.16  Bcast:10.53.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b25a:daff:fe02:1020/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5037083366 errors:0 dropped:100 overruns:0 frame:0
          TX packets:5068594045 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:835697551734 (778.3 GiB)  TX bytes:886055918384 (825.2 GiB)

So I expect the packet to be flooded to all vxlan tunnels, ports 1 and 4-7 (and seems like it's also supposed to output it back to port 8 for some reason). However, when I tcpdump -i eth1, I don't see the BOOTP packets vxlan-encapsulated (don't see the packets at all).

A similar second compute, running OVS 2.3.0 with no DPDK, managed by the same ODL, doesn't have this issue.

What can cause this alleged packet drop and how can I debug it?

ovs (with --dpdk) doesn't forward packets to vxlan tunnels in an openstack environment

I have a compute host with Debian 8, OVS 2.4.90, compiled for dpdk 2.0.0 and running with dpdk support (--dpdk), in an openstack environment with ODL on the controller as the OVS manager (so no neutron agent). A VM is created on the same compute I mentioned. The VM tries to obtain an IP from dhcp. It sends BOOTP packets.

Those packets are matched against:

cookie=0x0, duration=575.331s, table=110, n_packets=84, n_bytes=16056, priority=16383,reg0=0x1,tun_id=0xc9,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:8,output:4,output:1,output:5,output:6,output:7

This is how the bridge looks like:

OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:000072b0e2b1ed4c
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(vxlan-10.53.6.1): addr:7e:f9:5f:c9:86:6a
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-ext): addr:96:ee:48:d0:58:23
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(vxlan-10.53.6.1): addr:e2:f9:b2:7b:47:9f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(vxlan-10.53.6.1): addr:8e:26:fc:b8:b9:cb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 6(vxlan-10.53.6.1): addr:6a:69:b6:9a:47:56
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 7(vxlan-10.53.6.1): addr:de:ed:fc:3f:7c:c3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 8(tap79cdb725-73): addr:fe:16:3e:b0:bc:10
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:72:b0:e2:b1:ed:4c
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0

The local_ip configured on the bridge:

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-vsctl get Open_vSwitch . other_config:local_ip
"10.53.6.16"

The interface:

root@BASE-CCP-CPN-N0003-NETCLM:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr b0:5a:da:02:10:20
          inet addr:10.53.6.16  Bcast:10.53.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b25a:daff:fe02:1020/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5037083366 errors:0 dropped:100 overruns:0 frame:0
          TX packets:5068594045 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:835697551734 (778.3 GiB)  TX bytes:886055918384 (825.2 GiB)

So I expect the packet to be flooded to all vxlan tunnels, ports 1 and 4-7 (and seems like it's also supposed to output it back to port 8 for some reason). However, when I tcpdump -i eth1, I don't see the BOOTP packets vxlan-encapsulated (don't see the packets at all).

A similar second compute, running OVS 2.3.0 with no DPDK, managed by the same ODL, doesn't have this issue.

I see some different output between ovs-dpctl on those 2 computes:

The first one (the one with ovs-dpdk, where forwarding is not ok):

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-dpctl show
system@ovs-system:
        lookups: hit:10136371583 missed:5790438 lost:62027
        flows: 0
        masks: hit:73037709340 total:0 hit/pkt:7.20
        port 0: ovs-system (internal)

The second one (where forwarding is ok):

root@BASE-CCP-CPN-N0002-NETCLM:~# ovs-dpctl show
system@ovs-system:
        lookups: hit:9363817118 missed:7703922 lost:13217
        flows: 1
        masks: hit:73228225412 total:1 hit/pkt:7.81
        port 0: ovs-system (internal)
        port 1: br-int (internal)
        port 2: br-ex (internal)
        port 3: eth0.535
        port 4: vxlan_sys_4789 (vxlan: df_default=false, ttl=0)
        port 5: tapecf6958d-13

Does this mean the datapath wasn't created correctly on the on with ovs-dpdk? What can cause this alleged packet drop and how can I debug it?

ovs (with --dpdk) doesn't forward packets to vxlan tunnels in an openstack environment

I have a compute host with Debian 8, OVS 2.4.90, compiled for dpdk 2.0.0 and running with dpdk support (--dpdk), in an openstack environment with ODL on the controller as the OVS manager (so no neutron agent). A VM is created on the same compute I mentioned. The VM tries to obtain an IP from dhcp. It sends BOOTP packets.

Those packets are matched against:

cookie=0x0, duration=575.331s, table=110, n_packets=84, n_bytes=16056, priority=16383,reg0=0x1,tun_id=0xc9,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:8,output:4,output:1,output:5,output:6,output:7

This is how the bridge looks like:

OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:000072b0e2b1ed4c
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(vxlan-10.53.6.1): addr:7e:f9:5f:c9:86:6a
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-ext): addr:96:ee:48:d0:58:23
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(vxlan-10.53.6.1): addr:e2:f9:b2:7b:47:9f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(vxlan-10.53.6.1): addr:8e:26:fc:b8:b9:cb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 6(vxlan-10.53.6.1): addr:6a:69:b6:9a:47:56
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 7(vxlan-10.53.6.1): addr:de:ed:fc:3f:7c:c3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 8(tap79cdb725-73): addr:fe:16:3e:b0:bc:10
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:72:b0:e2:b1:ed:4c
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0

The local_ip configured on the bridge:

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-vsctl get Open_vSwitch . other_config:local_ip
"10.53.6.16"

The interface:

root@BASE-CCP-CPN-N0003-NETCLM:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr b0:5a:da:02:10:20
          inet addr:10.53.6.16  Bcast:10.53.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b25a:daff:fe02:1020/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5037083366 errors:0 dropped:100 overruns:0 frame:0
          TX packets:5068594045 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:835697551734 (778.3 GiB)  TX bytes:886055918384 (825.2 GiB)

So I expect the packet to be flooded to all vxlan tunnels, ports 1 and 4-7 (and seems like it's also supposed to output it back to port 8 for some reason). However, when I tcpdump -i eth1, I don't see the BOOTP packets vxlan-encapsulated (don't (I see the other non-related packets at all).vxlan-encapsulated such as lldp).

A similar second compute, running OVS 2.3.0 with no DPDK, managed by the same ODL, doesn't have this issue.

I see some different output between ovs-dpctl on those 2 computes:

The first one (the one with ovs-dpdk, where forwarding is not ok):

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-dpctl show
system@ovs-system:
        lookups: hit:10136371583 missed:5790438 lost:62027
        flows: 0
        masks: hit:73037709340 total:0 hit/pkt:7.20
        port 0: ovs-system (internal)

The second one (where forwarding is ok):

root@BASE-CCP-CPN-N0002-NETCLM:~# ovs-dpctl show
system@ovs-system:
        lookups: hit:9363817118 missed:7703922 lost:13217
        flows: 1
        masks: hit:73228225412 total:1 hit/pkt:7.81
        port 0: ovs-system (internal)
        port 1: br-int (internal)
        port 2: br-ex (internal)
        port 3: eth0.535
        port 4: vxlan_sys_4789 (vxlan: df_default=false, ttl=0)
        port 5: tapecf6958d-13

Does this mean the datapath wasn't created correctly on the on with ovs-dpdk? What can cause this alleged packet drop and how can I debug it?

ovs (with --dpdk) doesn't forward packets to vxlan tunnels in an openstack environment

I have a compute host with Debian 8, OVS 2.4.90, compiled for dpdk 2.0.0 and running with dpdk support (--dpdk), in an openstack environment with ODL on the controller as the OVS manager (so no neutron agent). A VM is created on the same compute I mentioned. The VM tries to obtain an IP from dhcp. It sends BOOTP packets.

Those packets are matched against:

cookie=0x0, duration=575.331s, table=110, n_packets=84, n_bytes=16056, priority=16383,reg0=0x1,tun_id=0xc9,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:8,output:4,output:1,output:5,output:6,output:7

This is how the bridge looks like:

OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:000072b0e2b1ed4c
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(vxlan-10.53.6.1): addr:7e:f9:5f:c9:86:6a
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-ext): addr:96:ee:48:d0:58:23
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(vxlan-10.53.6.1): addr:e2:f9:b2:7b:47:9f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(vxlan-10.53.6.1): addr:8e:26:fc:b8:b9:cb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 6(vxlan-10.53.6.1): addr:6a:69:b6:9a:47:56
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 7(vxlan-10.53.6.1): addr:de:ed:fc:3f:7c:c3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 8(tap79cdb725-73): addr:fe:16:3e:b0:bc:10
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:72:b0:e2:b1:ed:4c
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0

The local_ip configured on the bridge:

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-vsctl get Open_vSwitch . other_config:local_ip
"10.53.6.16"

The interface:

root@BASE-CCP-CPN-N0003-NETCLM:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr b0:5a:da:02:10:20
          inet addr:10.53.6.16  Bcast:10.53.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b25a:daff:fe02:1020/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5037083366 errors:0 dropped:100 overruns:0 frame:0
          TX packets:5068594045 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:835697551734 (778.3 GiB)  TX bytes:886055918384 (825.2 GiB)

So I expect the packet to be flooded to all vxlan tunnels, ports 1 and 4-7 (and seems like it's also supposed to output it back to port 8 for some reason). However, when I tcpdump -i eth1, I don't see the BOOTP packets vxlan-encapsulated (I see other non-related packets vxlan-encapsulated such as lldp).(or even not encapsulated for that matter).

A similar second compute, running OVS 2.3.0 with no DPDK, managed by the same ODL, doesn't have this issue.

I see some different output between ovs-dpctl on those 2 computes:

The first one (the one with ovs-dpdk, where forwarding is not ok):

root@BASE-CCP-CPN-N0003-NETCLM:~# ovs-dpctl show
system@ovs-system:
        lookups: hit:10136371583 missed:5790438 lost:62027
        flows: 0
        masks: hit:73037709340 total:0 hit/pkt:7.20
        port 0: ovs-system (internal)

The second one (where forwarding is ok):

root@BASE-CCP-CPN-N0002-NETCLM:~# ovs-dpctl show
system@ovs-system:
        lookups: hit:9363817118 missed:7703922 lost:13217
        flows: 1
        masks: hit:73228225412 total:1 hit/pkt:7.81
        port 0: ovs-system (internal)
        port 1: br-int (internal)
        port 2: br-ex (internal)
        port 3: eth0.535
        port 4: vxlan_sys_4789 (vxlan: df_default=false, ttl=0)
        port 5: tapecf6958d-13

Does this mean the datapath wasn't created correctly on the on with ovs-dpdk? What can cause this alleged packet drop and how can I debug it?