Errors to connect tcp:127.0.0.1:6640 and timeout on tcp:127.0.0.1:6633
Hi, I have a medium scenario running some OpenvSwitch Bridges and some Routers. It works, but I have some problems. :)
Scenario:
- I have 72 bridges connected to BR-INT (where Neutron control the VLAN translates UP/DOWN direction).
- I have 1 vRouters (root vRouter) with LAN (NET+SUBNET) for each of theses 72 Bridges.
- I create other vRouters connected to the LAN of the "root vRouter".
Everything works fine, but when I create more vRouterts (bellow root vRouters), problemas with neutron-openvswitch-agent or ovsdb-server. The problems are connections problems for port 6640 and 6633 on localhost.
I'm using Pike (installed by Packstack). On Queens it happens too, but not so easy.
I have 1 Controller and 2 Networks nodes.
On Controller, the only problem seems to be the remote neutron-openvswitch-agent down.
These logs bellow are from one of the Network nodes. If I restart the neutron-openvswitch-agent on this Network node, normally it works again.
Anybody knows if there a limit for bridges? I changes some timeouts for rpc, ovsdb, increased linux "open files" for openvswitch-agent PID(default 1024/4096), ... Where can I investigate more? Some Idea?
2018-08-30 08:25:29.965 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-736a8da0-fe3d-45dc-839b-4fe020b8208e - - - - -] Switch connection timeout: RuntimeError: ofctl request version=0x4,msg_type=0xe,msg_len=0x40,xid=0x6ff86b1,OFPFlowMod(buffer_id=4294967295,command=3,cookie=3807028968188950860L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'in_port': 2689}),out_group=4294967295,out_port=4294967295,priority=0,table_id=24) timed out
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-736a8da0-fe3d-45dc-839b-4fe020b8208e - - - - -] Error while processing VIF ports: RuntimeError: Switch connection timeout
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2066, in rpc_loop
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ofport_changed_ports = self.update_stale_ofport_rules()
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1210, in update_stale_ofport_rules
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.int_br.delete_arp_spoofing_protection(port=ofport)
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py", line 255, in delete_arp_spoofing_protection
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent match=match)
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File ...