Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

l3-agent errors in DVR configuration

We have an RDO/packstack installed Pike Controller/Neutron node and 7 Compute nodes that we are trying to configure using DVR. Following various guides I have placed configs in place but the l3agent on Controller is currently logging errors non-stop in the form of:

2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 00bfd2a0-c804-4b25-8f2a-30992fa6aa2e: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-fedcd71c-d8" does not exist.
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 538, in _process_router_update
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 475, in _process_router_if_compatible
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 490, in _process_updated_router
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     ri.process()
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 750, in process
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     self.fip_ns.scan_fip_ports(self)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 449, in scan_fip_ports
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     if len(ri.get_router_cidrs(device)):
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 288, in get_router_cidrs
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     centralized_cidrs = self._get_centralized_fip_cidr_set()
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 288, in get_router_cidrs
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     centralized_cidrs = self._get_centralized_fip_cidr_set()
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 274, in _get_centralized_fip_cidr_set
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     return set([addr['cidr'] for addr in device.addr.list()])
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 653, in list
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     self.name, scope, to, filters, ip_version)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 621, in get_devices_with_ip
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     for line in self._run(options, tuple(args)).split('\n'):
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _run
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     return self._parent._run(options, self.COMMAND, args)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 85, in _run
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     return self._as_root(options, command, args)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 100, in _as_root
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 109, in _execute
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent     raise ProcessExecutionError(msg, returncode=returncode)
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-fedcd71c-d8" does not exist.
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent
2018-02-11 01:03:16.239 31742 ERROR neutron.agent.l3.agent
2018-02-11 01:03:16.608 31742 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-fedcd71c-d8" does not exist

The errors are for each of the routers that exist in the environment and that have gateway set on external network. If I clear the gateway, the errors stop. If I add the gw they restart. The log for agent is growing, currently at 800mb The qg device that its complaining about does exist on compute nodes where the instances are running for that project, but not on the Controller.

=============Configuration

======Controller:

>>>>>>>>>>>>>>>>>>>>>>>>cat l3_agent.ini
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
agent_mode=dvr_snat
external_network_bridge =
debug=False

 >>>>>>>>>>>>>>>>>>>>>>>>cat neutron.conf
[DEFAULT]
bind_host=0.0.0.0
auth_strategy=keystone
core_plugin=neutron.plugins.ml2.plugin.Ml2Plugin
service_plugins=router
dns_domain = localnet
allow_overlapping_ips=True
router_scheduler_driver=neutron.scheduler.l3_agent_scheduler.ChanceScheduler
log_dir=/var/log/neutron
transport_url=rabbit://guest:guest@172.16.100.10:5672/
control_exchange=neutron
router_distributed = True
global_physnet_mtu=1550

[agent]
root_helper=sudo neutron-rootwrap /etc/neutron/rootwrap.conf

[cors]

[database]
connection=mysql+pymysql://neutron:password@172.16.100.10/neutron

[keystone_authtoken]
auth_uri=http://172.16.100.10:5000/
memcached_servers = localhost:11211
token_cache_time = 3000
revocation_cache_time = 600
auth_type=password
auth_url=http://172.16.100.10:35357
username=neutron
password=password
user_domain_name=Default
project_name=services
project_domain_name=Default

[matchmaker_redis]

[nova]
region_name=RegionOne
auth_url=http://172.16.100.10:35357
auth_type=password
password=password
project_domain_id=default
project_domain_name=Default
project_name=services
tenant_name=services
user_domain_id=default
user_domain_name=Default
username=nova

[oslo_concurrency]
lock_path=$state_path/lock

[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]

[oslo_messaging_rabbit]
ssl=False

[oslo_messaging_zmq]
[oslo_middleware]

[oslo_policy]
policy_file=/etc/neutron/policy.json

[quotas]
[ssl]
[service_providers]


>>>>>>>>>>>>>>>>>>>>>>>>cat plugins/ml2/ml2_conf.ini
[DEFAULT]

[ml2]
type_drivers=flat,vxlan
tenant_network_types=vxlan
mechanism_drivers=openvswitch,l2population
extension_drivers=port_security
path_mtu=1550
physical_network_mtus=extnet:1500

[ml2_type_flat]
flat_networks=*

[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]

[ml2_type_vxlan]
vni_ranges=1001:2000
vxlan_group=239.1.1.2

[securitygroup]
firewall_driver=neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver


>>>>>>>>>>>>>>>>>>>>>>>>cat plugins/ml2/openvswitch_agent.ini
[DEFAULT]

[agent]
tunnel_types=vxlan
enable_distributed_routing=True
l2_population = True
arp_responder = True

[ovs]
local_ip=172.16.100.10
bridge_mappings=extnet:br-ex

[securitygroup]
firewall_driver=neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

=======Compute:

>>>>>>>>>>>>>>>>>>>>>>>>cat l3_agent.ini 
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
agent_mode=dvr
external_network_bridge =
cat neutron.conf
[DEFAULT]
bind_host=0.0.0.0
auth_strategy=keystone
core_plugin=neutron.plugins.ml2.plugin.Ml2Plugin
allow_overlapping_ips=True
router_scheduler_driver=neutron.scheduler.l3_agent_scheduler.ChanceScheduler
log_dir=/var/log/neutron
transport_url=rabbit://guest:guest@172.16.100.10:5672/
service_plugins=router
global_physnet_mtu=1550

[agent]
root_helper=sudo neutron-rootwrap /etc/neutron/rootwrap.conf

[cors]

[database]
connection=mysql+pymysql://neutron:password@172.16.100.10/neutron

[keystone_authtoken]
auth_uri=http://172.16.100.10:5000/
auth_type=password
auth_url=http://172.16.100.10:35357
username=neutron
password=password
user_domain_name=Default
project_name=services
project_domain_name=Default

[matchmaker_redis]

[nova]
region_name=RegionOne
auth_url=http://172.16.100.10:35357
auth_type=password
password=password
project_domain_id=default
project_domain_name=Default
project_name=services
tenant_name=services
user_domain_id=default
user_domain_name=Default
username=nova

[oslo_concurrency]
lock_path=$state_path/lock

[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]

[oslo_messaging_rabbit]
ssl=False

[oslo_messaging_zmq]
[oslo_middleware]

[oslo_policy]
policy_file=/etc/neutron/policy.json

[quotas]
[ssl]
[service_providers]

>>>>>>>>>>>>>>>>>>>>>>>>cat ml2_conf.ini
[DEFAULT]

[ml2]
type_drivers=flat,vxlan
tenant_network_types=vxlan
mechanism_drivers=openvswitch,l2population
extension_drivers=port_security
path_mtu=1550
physical_network_mtus=extnet:1500

[ml2_type_flat]
flat_networks=*

[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]

[ml2_type_vxlan]
vni_ranges=1001:2000
vxlan_group=239.1.1.2

[securitygroup]
firewall_driver=neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

>>>>>>>>>>>>>>>>>>>>>>>>cat openvswitch_agent.ini
[DEFAULT]

[agent]
tunnel_types=vxlan
enable_distributed_routing=True
l2_population = True
arp_responder = True

[ovs]
local_ip=172.16.100.11
bridge_mappings=extnet:br-ex

[securitygroup]
firewall_driver=neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Some commands:

Controller:

ip netns list
qdhcp-1c137d04-735b-4572-9e0e-c38d18c2eb7d
snat-58eb954d-cd2b-4ed8-bce2-4f483749e1f8
qrouter-58eb954d-cd2b-4ed8-bce2-4f483749e1f8
qdhcp-53a91049-1784-4f6d-9c60-b2360fbd341d
snat-ea1664e4-e8fb-4607-b3ed-3a2fe3069c0b
qrouter-ea1664e4-e8fb-4607-b3ed-3a2fe3069c0b
snat-c1e9e6c4-b88c-446a-903f-62942a5cf856
qrouter-c1e9e6c4-b88c-446a-903f-62942a5cf856
qdhcp-5027730b-097e-4176-9246-395cbb97dad6
snat-6cd0f0cc-716a-4de3-b752-76d60b33f07e
qrouter-6cd0f0cc-716a-4de3-b752-76d60b33f07e
qdhcp-c5a33c16-bcf1-4357-b6f5-df7be4195e40
snat-00bfd2a0-c804-4b25-8f2a-30992fa6aa2e
qrouter-00bfd2a0-c804-4b25-8f2a-30992fa6aa2e
qdhcp-683e7a05-13d4-4166-94cd-97c2958d3427
snat-835782b5-031e-4551-a72d-10a88ed59e5e
qrouter-835782b5-031e-4551-a72d-10a88ed59e5e
qdhcp-7a9ab95c-992c-4277-890f-2e858d2e7325
snat-51ed2959-2fda-4df9-b7a1-83d3dda484b2
qrouter-51ed2959-2fda-4df9-b7a1-83d3dda484b2
qdhcp-8a1d1064-edd5-478a-8393-a1cd31602187
snat-6166f99e-cd8e-4225-a22b-14bc31c952f2
qrouter-6166f99e-cd8e-4225-a22b-14bc31c952f2
qdhcp-3e5bd10e-8019-4e65-8fda-5150fc293e13
fip-a8e7c94d-862d-41d4-9f59-ec7db567b868

for i in  1 2 3 4 5 7 8;do echo NODE$i;ssh NODE$i ip netns list;done
NODE1
qrouter-51ed2959-2fda-4df9-b7a1-83d3dda484b2
fip-a8e7c94d-862d-41d4-9f59-ec7db567b868
NODE2
fip-a8e7c94d-862d-41d4-9f59-ec7db567b868
qrouter-6166f99e-cd8e-4225-a22b-14bc31c952f2
qrouter-c1e9e6c4-b88c-446a-903f-62942a5cf856
qrouter-51ed2959-2fda-4df9-b7a1-83d3dda484b2
NODE3
NODE4
qrouter-6166f99e-cd8e-4225-a22b-14bc31c952f2
qrouter-51ed2959-2fda-4df9-b7a1-83d3dda484b2
fip-a8e7c94d-862d-41d4-9f59-ec7db567b868
NODE5
qrouter-6166f99e-cd8e-4225-a22b-14bc31c952f2
fip-a8e7c94d-862d-41d4-9f59-ec7db567b868
qrouter-51ed2959-2fda-4df9-b7a1-83d3dda484b2
NODE7
NODE8

Controller:

 ovs-vsctl show
    da3688e7-d587-41b9-a346-09c8fe7169f8
        Manager "ptcp:6640:127.0.0.1"
            is_connected: true
        Bridge br-tun
            Controller "tcp:127.0.0.1:6633"
                is_connected: true
            fail_mode: secure
            Port "vxlan-ac106411"
                Interface "vxlan-ac106411"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.17"}
            Port br-tun
                Interface br-tun
                    type: internal
            Port "vxlan-ac10640e"
                Interface "vxlan-ac10640e"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.14"}
            Port patch-int
                Interface patch-int
                    type: patch
                    options: {peer=patch-tun}
            Port "vxlan-ac10640f"
                Interface "vxlan-ac10640f"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.15"}
            Port "vxlan-ac10640d"
                Interface "vxlan-ac10640d"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.13"}
            Port "vxlan-ac106412"
                Interface "vxlan-ac106412"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.18"}
            Port "vxlan-ac10640b"
                Interface "vxlan-ac10640b"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.11"}
            Port "vxlan-ac10640c"
                Interface "vxlan-ac10640c"
                    type: vxlan
                    options: {df_default="true", in_key=flow,
    local_ip="172.16.100.10",
    out_key=flow,
    remote_ip="172.16.100.12"}
        Bridge br-ex
            Controller "tcp:127.0.0.1:6633"
                is_connected: true
            fail_mode: secure
            Port phy-br-ex
                Interface phy-br-ex
                    type: patch
                    options: {peer=int-br-ex}
            Port br-ex
                Interface br-ex
                    type: internal
            Port "eth0"
                Interface "eth0"
        Bridge br-int
            Controller "tcp:127.0.0.1:6633"
                is_connected: true
            fail_mode: secure
            Port "tapb0388621-cf"
                tag: 8
                Interface "tapb0388621-cf"
                    type: internal
            Port patch-tun
                Interface patch-tun
                    type: patch
                    options: {peer=patch-int}
            Port "tapc27910a7-5a"
                tag: 4
                Interface "tapc27910a7-5a"
                    type: internal
            Port "tap18d54170-e4"
                tag: 1
                Interface "tap18d54170-e4"
                    type: internal
            Port "tap738714ed-17"
                tag: 5
                Interface "tap738714ed-17"
                    type: internal
            Port "tap78e8ca0f-a2"
                tag: 7
                Interface "tap78e8ca0f-a2"
                    type: internal
            Port "tapb302c19f-05"
                tag: 6
                Interface "tapb302c19f-05"
                    type: internal
            Port "tapd6e16afb-6d"
                tag: 2
                Interface "tapd6e16afb-6d"
                    type: internal
            Port int-br-ex
                Interface int-br-ex
                    type: patch
                    options: {peer=phy-br-ex}
            Port "tap54535341-5c"
                tag: 3
                Interface "tap54535341-5c"
                    type: internal
            Port br-int
                Interface br-int
                    type: internal
        ovs_version: "2.7.3"

Compute

for i in  1 2 3 4 5 7 8;do echo NODE$i;ssh host-$i ovs-vsctl show;done
NODE1
8a178831-f1a1-43aa-b18f-4ede45f8e66c
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-5c22487e-e9"
            tag: 1
            Interface "qr-5c22487e-e9"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "fg-759bae0a-7c"
            tag: 2
            Interface "fg-759bae0a-7c"
                type: internal
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "eth0"
            Interface "eth0"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "vxlan-ac10640a"
            Interface "vxlan-ac10640a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.11", out_key=flow, remote_ip="172.16.100.10"}
        Port "vxlan-ac10640c"
            Interface "vxlan-ac10640c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.11", out_key=flow, remote_ip="172.16.100.12"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-ac10640f"
            Interface "vxlan-ac10640f"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.11", out_key=flow, remote_ip="172.16.100.15"}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-ac10640e"
            Interface "vxlan-ac10640e"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.11", out_key=flow, remote_ip="172.16.100.14"}
    ovs_version: "2.7.3"
NODE2
151c03c5-8848-4fe9-87ed-3348b1a41de8
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qr-5c22487e-e9"
            tag: 2
            Interface "qr-5c22487e-e9"
                type: internal
        Port "fg-007681cd-cf"
            tag: 4
            Interface "fg-007681cd-cf"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-3fda2fad-af"
            tag: 1
            Interface "qr-3fda2fad-af"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "qr-27f884cf-94"
            tag: 3
            Interface "qr-27f884cf-94"
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-ac10640a"
            Interface "vxlan-ac10640a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.12", out_key=flow, remote_ip="172.16.100.10"}
        Port "vxlan-ac10640f"
            Interface "vxlan-ac10640f"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.12", out_key=flow, remote_ip="172.16.100.15"}
        Port "vxlan-ac10640b"
            Interface "vxlan-ac10640b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.12", out_key=flow, remote_ip="172.16.100.11"}
        Port "vxlan-ac10640e"
            Interface "vxlan-ac10640e"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.12", out_key=flow, remote_ip="172.16.100.14"}
        Port "vxlan-ac10640d"
            Interface "vxlan-ac10640d"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.12", out_key=flow, remote_ip="172.16.100.13"}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth0"
            Interface "eth0"
    ovs_version: "2.7.3"
NODE3
218f05d7-ed0b-49fc-a160-b1adda44a7aa
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "eth0"
            Interface "eth0"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    ovs_version: "2.7.3"
NODE4
5258a367-f187-41a0-8f10-6fb80b027706
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "fg-52b94249-91"
            tag: 3
            Interface "fg-52b94249-91"
                type: internal
        Port "qr-5c22487e-e9"
            tag: 1
            Interface "qr-5c22487e-e9"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-27f884cf-94"
            tag: 2
            Interface "qr-27f884cf-94"
                type: internal
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth0"
            Interface "eth0"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-ac10640f"
            Interface "vxlan-ac10640f"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.14", out_key=flow, remote_ip="172.16.100.15"}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-ac10640a"
            Interface "vxlan-ac10640a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.14", out_key=flow, remote_ip="172.16.100.10"}
        Port "vxlan-ac10640b"
            Interface "vxlan-ac10640b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.14", out_key=flow, remote_ip="172.16.100.11"}
        Port "vxlan-ac10640c"
            Interface "vxlan-ac10640c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.14", out_key=flow, remote_ip="172.16.100.12"}
    ovs_version: "2.7.3"
NODE5
ddcac88a-df8e-4d15-b17e-fa68c138de15
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "vxlan-ac10640b"
            Interface "vxlan-ac10640b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.15", out_key=flow, remote_ip="172.16.100.11"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-ac10640e"
            Interface "vxlan-ac10640e"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.15", out_key=flow, remote_ip="172.16.100.14"}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-ac10640a"
            Interface "vxlan-ac10640a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.15", out_key=flow, remote_ip="172.16.100.10"}
        Port "vxlan-ac10640c"
            Interface "vxlan-ac10640c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.100.15", out_key=flow, remote_ip="172.16.100.12"}
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-5c22487e-e9"
            tag: 1
            Interface "qr-5c22487e-e9"
                type: internal
        Port "qr-27f884cf-94"
            tag: 2
            Interface "qr-27f884cf-94"
                type: internal
        Port "fg-682c7f3e-b0"
            tag: 3
            Interface "fg-682c7f3e-b0"
                type: internal
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth0"
            Interface "eth0"
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.7.3"
NODE7
f56d4c32-521e-44ee-a5e1-a94effcf7da6
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.7.3"
NODE8
1804cd9e-4ac1-4ec8-82a8-5fddbcfb578a
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.7.3"






[root@controler neutron]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x6f5874ddc05da9ff, duration=4994.705s, table=0, n_packets=11957, n_bytes=1690773, idle_age=4990, priority=2,in_port=2 actions=resubmit(,1)
 cookie=0x6f5874ddc05da9ff, duration=5039.188s, table=0, n_packets=14781, n_bytes=2895605, idle_age=65534, priority=0 actions=NORMAL
 cookie=0x6f5874ddc05da9ff, duration=4994.703s, table=0, n_packets=37110676, n_bytes=46767171442, idle_age=0, priority=1 actions=resubmit(,3)
 cookie=0x6f5874ddc05da9ff, duration=4994.702s, table=1, n_packets=11957, n_bytes=1690773, idle_age=4990, priority=0 actions=resubmit(,2)
 cookie=0x6f5874ddc05da9ff, duration=4994.702s, table=2, n_packets=11957, n_bytes=1690773, idle_age=4990, priority=2,in_port=2 actions=drop
 cookie=0x6f5874ddc05da9ff, duration=4994.612s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:1a:53:e5 actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.606s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:20:0d:15 actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.602s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:2a:d8:c7 actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.596s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:4c:b7:5c actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.591s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:63:56:00 actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.586s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:91:75:9e actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.580s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=2,dl_src=fa:16:3f:fe:b1:24 actions=output:2
 cookie=0x6f5874ddc05da9ff, duration=4994.700s, table=3, n_packets=37110675, n_bytes=46767171376, idle_age=0, priority=1 actions=NORMAL
[root@controler neutron]# 
[root@controler neutron]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x41917c1c0e7cdd99, duration=5001.021s, table=0, n_packets=411077, n_bytes=46664111, idle_age=4996, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0x41917c1c0e7cdd99, duration=4988.051s, table=0, n_packets=10, n_bytes=660, idle_age=3584, priority=1,in_port=3 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=4988.047s, table=0, n_packets=0, n_bytes=0, idle_age=4988, priority=1,in_port=18 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=4988.043s, table=0, n_packets=0, n_bytes=0, idle_age=4988, priority=1,in_port=2 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=4988.029s, table=0, n_packets=0, n_bytes=0, idle_age=4988, priority=1,in_port=24 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=4988.025s, table=0, n_packets=10, n_bytes=660, idle_age=4174, priority=1,in_port=5 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=4988.020s, table=0, n_packets=0, n_bytes=0, idle_age=4988, priority=1,in_port=6 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=4988.016s, table=0, n_packets=0, n_bytes=0, idle_age=4988, priority=1,in_port=20 actions=resubmit(,4)
 cookie=0x41917c1c0e7cdd99, duration=5045.477s, table=0, n_packets=1806, n_bytes=379762, idle_age=8186, priority=0 actions=drop
 cookie=0x41917c1c0e7cdd99, duration=5001.020s, table=1, n_packets=410657, n_bytes=46629402, idle_age=4996, priority=0 actions=resubmit(,2)
 cookie=0x41917c1c0e7cdd99, duration=5045.476s, table=2, n_packets=304272, n_bytes=18228978, idle_age=13145, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21)
 cookie=0x41917c1c0e7cdd99, duration=5045.475s, table=2, n_packets=3097, n_bytes=512797, idle_age=36193, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x41917c1c0e7cdd99, duration=5045.473s, table=2, n_packets=103605, n_bytes=27918010, idle_age=4996, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x41917c1c0e7cdd99, duration=5045.472s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x41917c1c0e7cdd99, duration=4991.462s, table=4, n_packets=5017, n_bytes=446558, idle_age=46604, priority=1,tun_id=0x3ff actions=mod_vlan_vid:1,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.456s, table=4, n_packets=10, n_bytes=660, idle_age=4174, priority=1,tun_id=0x42a actions=mod_vlan_vid:2,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.450s, table=4, n_packets=0, n_bytes=0, idle_age=13189, priority=1,tun_id=0x43a actions=mod_vlan_vid:3,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.443s, table=4, n_packets=10, n_bytes=660, idle_age=3584, priority=1,tun_id=0x430 actions=mod_vlan_vid:4,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.436s, table=4, n_packets=0, n_bytes=0, idle_age=4991, priority=1,tun_id=0x436 actions=mod_vlan_vid:5,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.429s, table=4, n_packets=0, n_bytes=0, idle_age=13271, priority=1,tun_id=0x3f4 actions=mod_vlan_vid:6,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.418s, table=4, n_packets=0, n_bytes=0, idle_age=13226, priority=1,tun_id=0x3f2 actions=mod_vlan_vid:7,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=4991.412s, table=4, n_packets=0, n_bytes=0, idle_age=13145, priority=1,tun_id=0x3ea actions=mod_vlan_vid:8,resubmit(,9)
 cookie=0x41917c1c0e7cdd99, duration=5045.471s, table=4, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x41917c1c0e7cdd99, duration=5045.470s, table=6, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x41917c1c0e7cdd99, duration=5000.922s, table=9, n_packets=1314, n_bytes=115164, idle_age=43539, priority=1,dl_src=fa:16:3f:1a:53:e5 actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5000.917s, table=9, n_packets=1580, n_bytes=147738, idle_age=43552, priority=1,dl_src=fa:16:3f:20:0d:15 actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5000.912s, table=9, n_packets=1284, n_bytes=126560, idle_age=4174, priority=1,dl_src=fa:16:3f:2a:d8:c7 actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5000.906s, table=9, n_packets=1076, n_bytes=90790, idle_age=43570, priority=1,dl_src=fa:16:3f:4c:b7:5c actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5000.901s, table=9, n_packets=1137, n_bytes=106248, idle_age=43539, priority=1,dl_src=fa:16:3f:63:56:00 actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5000.896s, table=9, n_packets=1432, n_bytes=127485, idle_age=3584, priority=1,dl_src=fa:16:3f:91:75:9e actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5000.891s, table=9, n_packets=1344, n_bytes=123292, idle_age=43540, priority=1,dl_src=fa:16:3f:fe:b1:24 actions=output:1
 cookie=0x41917c1c0e7cdd99, duration=5001.020s, table=9, n_packets=10906, n_bytes=1621330, idle_age=13329, priority=0 actions=resubmit(,10)
 cookie=0x41917c1c0e7cdd99, duration=5045.468s, table=10, n_packets=10906, n_bytes=1621330, idle_age=13329, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x41917c1c0e7cdd99,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1
 cookie=0x41917c1c0e7cdd99, duration=4984.016s, table=20, n_packets=0, n_bytes=0, idle_age=4984, priority=2,dl_vlan=2,dl_dst=fa:16:3e:33:95:3d actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:3
 cookie=0x41917c1c0e7cdd99, duration=4984.013s, table=20, n_packets=0, n_bytes=0, idle_age=4984, priority=2,dl_vlan=2,dl_dst=fa:16:3e:9c:0a:f5 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:3
 cookie=0x41917c1c0e7cdd99, duration=4984.009s, table=20, n_packets=0, n_bytes=0, idle_age=4984, priority=2,dl_vlan=2,dl_dst=fa:16:3e:92:c8:03 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:2
 cookie=0x41917c1c0e7cdd99, duration=4984.006s, table=20, n_packets=0, n_bytes=0, idle_age=4984, priority=2,dl_vlan=2,dl_dst=fa:16:3e:82:3b:54 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:2
 cookie=0x41917c1c0e7cdd99, duration=4984.001s, table=20, n_packets=0, n_bytes=0, idle_age=4984, priority=2,dl_vlan=2,dl_dst=fa:16:3e:d4:12:65 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:5
 cookie=0x41917c1c0e7cdd99, duration=4983.999s, table=20, n_packets=0, n_bytes=0, idle_age=4983, priority=2,dl_vlan=2,dl_dst=fa:16:3e:65:fd:56 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:5
 cookie=0x41917c1c0e7cdd99, duration=4983.993s, table=20, n_packets=0, n_bytes=0, idle_age=4983, priority=2,dl_vlan=2,dl_dst=fa:16:3e:90:7c:04 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:6
 cookie=0x41917c1c0e7cdd99, duration=4976.796s, table=20, n_packets=0, n_bytes=0, idle_age=4976, priority=2,dl_vlan=4,dl_dst=fa:16:3e:13:c1:2f actions=strip_vlan,load:0x430->NXM_NX_TUN_ID[],output:3
 cookie=0x41917c1c0e7cdd99, duration=4976.788s, table=20, n_packets=0, n_bytes=0, idle_age=4976, priority=2,dl_vlan=4,dl_dst=fa:16:3e:b2:99:a4 actions=strip_vlan,load:0x430->NXM_NX_TUN_ID[],output:5
 cookie=0x41917c1c0e7cdd99, duration=4976.785s, table=20, n_packets=0, n_bytes=0, idle_age=4976, priority=2,dl_vlan=4,dl_dst=fa:16:3e:b8:dd:55 actions=strip_vlan,load:0x430->NXM_NX_TUN_ID[],output:5
 cookie=0x41917c1c0e7cdd99, duration=4976.782s, table=20, n_packets=0, n_bytes=0, idle_age=4976, priority=2,dl_vlan=4,dl_dst=fa:16:3e:ca:42:b5 actions=strip_vlan,load:0x430->NXM_NX_TUN_ID[],output:6
 cookie=0x41917c1c0e7cdd99, duration=4973.424s, table=20, n_packets=0, n_bytes=0, idle_age=4973, priority=2,dl_vlan=5,dl_dst=fa:16:3e:0a:4d:ae actions=strip_vlan,load:0x436->NXM_NX_TUN_ID[],output:3
 cookie=0x41917c1c0e7cdd99, duration=4973.420s, table=20, n_packets=0, n_bytes=0, idle_age=4973, priority=2,dl_vlan=5,dl_dst=fa:16:3e:e4:db:52 actions=strip_vlan,load:0x436->NXM_NX_TUN_ID[],output:3
 cookie=0x41917c1c0e7cdd99, duration=4973.418s, table=20, n_packets=0, n_bytes=0, idle_age=4973, priority=2,dl_vlan=5,dl_dst=fa:16:3e:39:e8:fe actions=strip_vlan,load:0x436->NXM_NX_TUN_ID[],output:3
 cookie=0x41917c1c0e7cdd99, duration=5045.467s, table=20, n_packets=176, n_bytes=12259, idle_age=65534, priority=0 actions=resubmit(,22)
 cookie=0x41917c1c0e7cdd99, duration=4984.017s, table=21, n_packets=0, n_bytes=0, idle_age=4984, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.6 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e33953d->NXM_NX_ARP_SHA[],load:0xa0a0a06->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:33:95:3d,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4984.014s, table=21, n_packets=0, n_bytes=0, idle_age=4984, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.18 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e9c0af5->NXM_NX_ARP_SHA[],load:0xa0a0a12->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:9c:0a:f5,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4984.010s, table=21, n_packets=0, n_bytes=0, idle_age=4984, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.10 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e92c803->NXM_NX_ARP_SHA[],load:0xa0a0a0a->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:92:c8:03,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4984.007s, table=21, n_packets=0, n_bytes=0, idle_age=4984, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.15 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e823b54->NXM_NX_ARP_SHA[],load:0xa0a0a0f->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:82:3b:54,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4984.003s, table=21, n_packets=0, n_bytes=0, idle_age=4984, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.14 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ed41265->NXM_NX_ARP_SHA[],load:0xa0a0a0e->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:d4:12:65,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4984s, table=21, n_packets=0, n_bytes=0, idle_age=4984, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.16 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e65fd56->NXM_NX_ARP_SHA[],load:0xa0a0a10->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:65:fd:56,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4983.996s, table=21, n_packets=0, n_bytes=0, idle_age=4983, priority=1,arp,dl_vlan=2,arp_tpa=10.10.10.12 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e907c04->NXM_NX_ARP_SHA[],load:0xa0a0a0c->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:90:7c:04,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4976.797s, table=21, n_packets=0, n_bytes=0, idle_age=4976, priority=1,arp,dl_vlan=4,arp_tpa=10.10.10.12 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e13c12f->NXM_NX_ARP_SHA[],load:0xa0a0a0c->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:13:c1:2f,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4976.789s, table=21, n_packets=0, n_bytes=0, idle_age=4976, priority=1,arp,dl_vlan=4,arp_tpa=10.10.10.4 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163eb299a4->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:b2:99:a4,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4976.786s, table=21, n_packets=0, n_bytes=0, idle_age=4976, priority=1,arp,dl_vlan=4,arp_tpa=10.10.10.6 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163eb8dd55->NXM_NX_ARP_SHA[],load:0xa0a0a06->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:b8:dd:55,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4976.783s, table=21, n_packets=0, n_bytes=0, idle_age=4976, priority=1,arp,dl_vlan=4,arp_tpa=10.10.10.10 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163eca42b5->NXM_NX_ARP_SHA[],load:0xa0a0a0a->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:ca:42:b5,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4973.425s, table=21, n_packets=0, n_bytes=0, idle_age=4973, priority=1,arp,dl_vlan=5,arp_tpa=10.10.10.11 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0a4dae->NXM_NX_ARP_SHA[],load:0xa0a0a0b->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0a:4d:ae,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4973.422s, table=21, n_packets=0, n_bytes=0, idle_age=4973, priority=1,arp,dl_vlan=5,arp_tpa=10.10.10.13 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ee4db52->NXM_NX_ARP_SHA[],load:0xa0a0a0d->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:e4:db:52,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=4973.419s, table=21, n_packets=0, n_bytes=0, idle_age=4973, priority=1,arp,dl_vlan=5,arp_tpa=10.10.10.9 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e39e8fe->NXM_NX_ARP_SHA[],load:0xa0a0a09->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:39:e8:fe,IN_PORT
 cookie=0x41917c1c0e7cdd99, duration=5045.466s, table=21, n_packets=304147, n_bytes=18223728, idle_age=13145, priority=0 actions=resubmit(,22)
 cookie=0x41917c1c0e7cdd99, duration=4988.014s, table=22, n_packets=0, n_bytes=0, idle_age=4988, priority=1,dl_vlan=1 actions=strip_vlan,load:0x3ff->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=4983.997s, table=22, n_packets=0, n_bytes=0, idle_age=4984, priority=1,dl_vlan=2 actions=strip_vlan,load:0x42a->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6
 cookie=0x41917c1c0e7cdd99, duration=4980.393s, table=22, n_packets=0, n_bytes=0, idle_age=4980, priority=1,dl_vlan=3 actions=strip_vlan,load:0x43a->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=4976.781s, table=22, n_packets=0, n_bytes=0, idle_age=4976, priority=1,dl_vlan=4 actions=strip_vlan,load:0x430->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=4973.410s, table=22, n_packets=0, n_bytes=0, idle_age=4973, priority=1,dl_vlan=5 actions=strip_vlan,load:0x436->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=4969.896s, table=22, n_packets=0, n_bytes=0, idle_age=4969, priority=1,dl_vlan=6 actions=strip_vlan,load:0x3f4->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=4966.024s, table=22, n_packets=0, n_bytes=0, idle_age=4966, priority=1,dl_vlan=7 actions=strip_vlan,load:0x3f2->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=4962.652s, table=22, n_packets=0, n_bytes=0, idle_age=4962, priority=1,dl_vlan=8 actions=strip_vlan,load:0x3ea->NXM_NX_TUN_ID[],output:2,output:3,output:5,output:6,output:18,output:20,output:24
 cookie=0x41917c1c0e7cdd99, duration=5045.465s, table=22, n_packets=407498, n_bytes=46125726, idle_age=4996, priority=0 actions=drop
[root@controler neutron]# 
[root@controler neutron]# 
[root@controler neutron]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x9b14495501a8ef89, duration=5005.714s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:1a:53:e5 actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.711s, table=0, n_packets=1314, n_bytes=115164, idle_age=43543, priority=2,in_port=2,dl_src=fa:16:3f:1a:53:e5 actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.709s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:20:0d:15 actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.706s, table=0, n_packets=1580, n_bytes=147738, idle_age=43556, priority=2,in_port=2,dl_src=fa:16:3f:20:0d:15 actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.704s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:2a:d8:c7 actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.701s, table=0, n_packets=1284, n_bytes=126560, idle_age=4179, priority=2,in_port=2,dl_src=fa:16:3f:2a:d8:c7 actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.698s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:4c:b7:5c actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.696s, table=0, n_packets=1076, n_bytes=90790, idle_age=43575, priority=2,in_port=2,dl_src=fa:16:3f:4c:b7:5c actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.693s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:63:56:00 actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.691s, table=0, n_packets=1137, n_bytes=106248, idle_age=43543, priority=2,in_port=2,dl_src=fa:16:3f:63:56:00 actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.688s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:91:75:9e actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.686s, table=0, n_packets=1432, n_bytes=127485, idle_age=3589, priority=2,in_port=2,dl_src=fa:16:3f:91:75:9e actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.683s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:fe:b1:24 actions=resubmit(,2)
 cookie=0x9b14495501a8ef89, duration=5005.680s, table=0, n_packets=1344, n_bytes=123292, idle_age=43545, priority=2,in_port=2,dl_src=fa:16:3f:fe:b1:24 actions=resubmit(,1)
 cookie=0x9b14495501a8ef89, duration=5005.810s, table=0, n_packets=522828, n_bytes=61823324, idle_age=0, priority=2,in_port=1 actions=drop
 cookie=0x9b14495501a8ef89, duration=5050.443s, table=0, n_packets=17616, n_bytes=2377665, idle_age=5001, priority=0 actions=resubmit(,60)
 cookie=0x9b14495501a8ef89, duration=5005.812s, table=1, n_packets=6168, n_bytes=525976, idle_age=3589, priority=1 actions=drop
 cookie=0x9b14495501a8ef89, duration=5005.811s, table=2, n_packets=0, n_bytes=0, idle_age=65534, priority=1 actions=drop
 cookie=0x9b14495501a8ef89, duration=5005.813s, table=23, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x9b14495501a8ef89, duration=5050.441s, table=24, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x9b14495501a8ef89, duration=5050.442s, table=60, n_packets=422803, n_bytes=48357286, idle_age=5001, priority=3 actions=NORMAL