Ask Your Question
0

Why would an OVS bridge not forward ARP

asked 2012-09-20 23:07:13 -0500

sunil-srivastava gravatar image

Bridge "br-eth0" Port "br-eth0" Interface "br-eth0" type: internal Port "eth0" Interface "eth0" Port "phy-br-eth0" Interface "phy-br-eth0" Bridge br-int Port "tap55d1e5e8-ab" tag: 1 Interface "tap55d1e5e8-ab" type: internal Port "qr-4b50a17d-3c" tag: 1 Interface "qr-4b50a17d-3c" type: internal Port "int-br-eth0" Interface "int-br-eth0" Port "tape8d6e0a5-52" tag: 1 Interface "tape8d6e0a5-52" Port "tap6176588e-48" tag: 1 Interface "tap6176588e-48" Port br-int Interface br-int type: internal

I can see ARP packets sent from int-br-eth0 to phy-br-eth0 but not to upstream eth0.

So we cannot ping from one VM (or DHCP NetNS) on one machine to another VM on another machine.

I see the ping triggering ARPs. The Tx counter of int-br-eth0 and Rx counter of phy-br-eth0 were also corelated with ping.

edit retag flag offensive close merge delete

25 answers

Sort by ยป oldest newest most voted
0

answered 2012-09-21 05:24:04 -0500

sunil-srivastava gravatar image

Hi Yong,

That is not the root cause. Still.

stack@esg-dell-c4-s11:~/gitstack/devstack$ quantum net-list +--------------------------------------+---------+--------------------------------------+ | id | name | subnets | +--------------------------------------+---------+--------------------------------------+ | 68f76ec1-407b-4e42-a089-d0e6553473f8 | ext_net | 09851d25-806f-492c-b708-bf03838d77b3 | | fa8f9c5e-e41a-4f80-955c-94b3a45b9dcb | net1 | 31ed889f-f3f5-4faa-bb51-1d92344c91a3 | +--------------------------------------+---------+--------------------------------------+

stack@esg-dell-c4-s11:~/gitstack/devstack$ quantum net-show net1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | fa8f9c5e-e41a-4f80-955c-94b3a45b9dcb | | name | net1 | | provider:network_type | local | | provider:physical_network | | | provider:segmentation_id | | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 31ed889f-f3f5-4faa-bb51-1d92344c91a3 | | tenant_id | b0d8717a0f8b4cf8bdff8d84156622af |

stack@esg-dell-c4-s11:~/gitstack/devstack$ quantum net-show ext_net +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | 68f76ec1-407b-4e42-a089-d0e6553473f8 | | name | ext_net | | provider:network_type | local | | provider:physical_network | | | provider:segmentation_id | | | router:external | True | | shared | False | | status | ACTIVE | | subnets | 09851d25-806f-492c-b708-bf03838d77b3 | | tenant_id | 44cb33fdc72b44ad8e200a1326199895 | +---------------------------+--------------------------------------+

+---------------------------+--------------------------------------+

edit flag offensive delete link more
0

answered 2012-09-21 06:10:35 -0500

Sunil, One last thing. If you leave the ping running and then provide the output of

ovs-dpctl dump-flows br-int ovs-dpctl dump-flows br-tun

That will show the active flow entires in the kernel. Did you try running tcpdump on eth0 to see if you see arp packets there? You never said how you know that they are not making it out eth0. You just said you were unable to ping. (The drop flow entry you provided blocks the returning replies so ping definitely will not work).

Aaron

edit flag offensive delete link more
0

answered 2012-09-21 06:26:42 -0500

sunil-srivastava gravatar image

Hi Aron,

Here is the TCP Dumps.

The following output show there is no link issue between phy-br-eth0 and int-br-eth0.

(1)

root@esg-dell-c4-s11:~# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data. 64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=0.056 ms 64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=0.052 ms 64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=0.032 ms 64 bytes from 10.0.0.2: icmp_req=4 ttl=64 time=0.041 ms 64 bytes from 10.0.0.2: icmp_req=5 ttl=64 time=0.048 ms --- 10.0.0.2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms

The above triggers this below on int-br-eth0

stack@esg-dell-c4-s11:~/devstack$ sudo tcpdump -i int-br-eth0 arp and src 10.0.0.2 tcpdump: WARNING: int-br-eth0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on int-br-eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 00:41:55.755545 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:41:56.753793 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:41:57.753782 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:41:58.771011 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:41:59.769790 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:42:00.769796 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28

(2)

And again on phy-br-eth0.

root@esg-dell-c4-s11:~# ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data. From 10.0.0.2 icmp_seq=1 Destination Host Unreachable From 10.0.0.2 icmp_seq=2 Destination Host Unreachable From 10.0.0.2 icmp_seq=3 Destination Host Unreachable ^C --- 10.0.0.3 ping statistics --- 5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4024ms

stack@esg-dell-c4-s11:~/devstack$ sudo tcpdump -i phy-br-eth0 arp and src 10.0.0.2 tcpdump: WARNING: phy-br-eth0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on phy-br-eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 00:43:35.871097 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:43:36.869787 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:43:37.873777 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length 28 00:43:38.887008 ARP, Request who-has http://inenbasavbl1c.corp.emc.com tell http://usxxstephc2mbp1.corp.emc.com , length ... (more)

edit flag offensive delete link more
0

answered 2012-09-21 06:43:55 -0500

Hi Sunil,

Sorry I'm not sure why you're trying to show me here. You can ping 10.0.0.2 and not 10.0.0.3, I don't know where those interfaces reside in your setup. Can you show me an ifconfig -a of this machine. Also while you are pinging a ovs-dpctl dump-flow. Also, why are you showing me a tcpdump on phy-br-eth0, you should be doing that on eth0 since you say the packets are getting there.

Thanks,

Aaron

P.S: I'll also be in #openstack-dev for a little while longer tonight.

edit flag offensive delete link more
0

answered 2012-09-21 07:01:47 -0500

sunil-srivastava gravatar image

Someone else has been on the setup and things have changed a bit.

He said he entered a flow entry - but I did not have time to follow up. I am in R/O mode. :-)

br-eth0 was brought up and that may have changed the behavior.

Now ARP is reach the other machine, and I can see the traffic on eth0, phy-br-eth0 and int-br-eth0.

But there is not ARP reply. The ARPs are not getting to any of the TAP interfaces (but I only did once).

The br-int seems to be dropping now.

BTW, one TAP interface is not there as VM was brought down but ovs-vsctl has it.

stack@esg-dell-c4-s10:~/devstack$ ovs-dpctl show br-eth0 flows system@br-eth0: lookups: hit:658100 missed:119465 lost:0 flows: 27 port 0: br-eth0 (internal) port 6: eth0 port 9: phy-br-eth0 ovs-dpctl: opening datapath flows failed (No such device)

stack@esg-dell-c4-s10:~/devstack$ sudo ovs-ofctl dump-flows br-eth0 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=54070.378s, table=0, n_packets=80, n_bytes=6552, priority=2,in_port=9 actions=drop cookie=0x0, duration=54070.663s, table=0, n_packets=632974, n_bytes=79155773, priority=1 actions=NORMAL

stack@esg-dell-c4-s10:~/devstack$ ovs-dpctl show br-eth0 flows system@br-eth0: lookups: hit:660125 missed:119838 lost:0 flows: 30 port 0: br-eth0 (internal) port 6: eth0 port 9: phy-br-eth0 ovs-dpctl: opening datapath flows failed (No such device)

stack@esg-dell-c4-s10:~/devstack$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=54307.045s, table=0, n_packets=608686, n_bytes=77906470, priority=2,in_port=18 actions=drop cookie=0x0, duration=54307.359s, table=0, n_packets=14395, n_bytes=2748282, priority=1 actions=NORMAL

stack@esg-dell-c4-s10:~/devstack$ ovs-dpctl show br-int flows system@br-int: lookups: hit:631616 missed:143682 lost:0 flows: 26 port 0: br-int (internal) Sep 21 03:00:32|00001|netdev_linux|WARN|/sys/class/net/tap26583155-34/carrier: open failed: No such file or directory port 1: tap26583155-34 (internal) port 14: tapd1802d22-b4 port 15: tapfa0e7fcf-8d port 16: tap5eb27feb-05 port 18: int-br-eth0 ovs-dpctl: opening datapath flows failed (No such device)

edit flag offensive delete link more
0

answered 2012-09-21 07:03:28 -0500

sunil-srivastava gravatar image

Sorry, I need to go. Someone from EMC would follow up.

edit flag offensive delete link more
0

answered 2012-09-21 07:03:31 -0500

sunil-srivastava gravatar image

Sorry, I need to go. Someone from EMC would follow up.

edit flag offensive delete link more
0

answered 2012-09-21 07:51:00 -0500

gongysh gravatar image

Seen from Floor #11, you are using local mode where the traffic will not go out from the machine. U can try multiple Vms on the same machine, they should can ping each other.

edit flag offensive delete link more
0

answered 2012-09-21 13:58:25 -0500

eoghank gravatar image

Following on from Yong's suggestion about flow control rules I checked the rules on br-int and br-eth0 on both nodes. The example below is from br-int on the controller and port 20 was set to drop by default. This was the same for br-int on the other node, and for br-eth0 on both nodes.

sudo ovs-ofctl show br-int OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:00005aa5a97a1541 n_tables:255, n_buffers:256 features: capabilities:0xc7, actions:0xfff 2(tap55d1e5e8-ab): addr:0b:02:00:00:00:00 config: PORT_DOWN state: LINK_DOWN 18(tape8d6e0a5-52): addr:b6:a6:52:18:de:00 config: 0 state: 0 current: 10MB-FD COPPER 19(tap6176588e-48): addr:7e:b5:44:0c:fa:0d config: 0 state: 0 current: 10MB-FD COPPER 20(int-br-eth0): addr:6a:36:6c:2e:3a:76 config: 0 state: 0 current: 10GB-FD COPPER

$sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=75826.521s, table=0, n_packets=810503, n_bytes=104709261, priority=2,in_port=20 actions=drop cookie=0x0, duration=75826.959s, table=0, n_packets=50413, n_bytes=9135002, priority=1 actions=NORMAL

Once I opened these up for both br-int and br-eth0 on both sides I could ping instances from either side so this is now working.

Is there any reason why these ports would be set to drop by default?

Thanks Eoghan

edit flag offensive delete link more
0

answered 2012-09-21 14:49:49 -0500

gongysh gravatar image

both of your networks are local type: stack@esg-dell-c4-s11:~/gitstack/devstack$ quantum net-show net1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | fa8f9c5e-e41a-4f80-955c-94b3a45b9dcb | | name | net1 | | provider:network_type | local |

the flow is to drop by default. if we have Vms on the network with Vlan network_type, the port will be opened.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-09-20 23:07:13 -0500

Seen: 2,509 times

Last updated: Sep 26 '12