Ask Your Question
0

Errors to connect tcp:127.0.0.1:6640 and timeout on tcp:127.0.0.1:6633

asked 2018-09-02 09:49:26 -0600

LuisKleber gravatar image

Hi, I have a medium scenario running some OpenvSwitch Bridges and some Routers. It works, but I have some problems. :)

Scenario:

  • I have 72 bridges connected to BR-INT (where Neutron control the VLAN translates UP/DOWN direction).
  • I have 1 vRouters (root vRouter) with LAN (NET+SUBNET) for each of theses 72 Bridges.
  • I create other vRouters connected to the LAN of the "root vRouter".

Everything works fine, but when I create more vRouterts (bellow root vRouters), problemas with neutron-openvswitch-agent or ovsdb-server. The problems are connections problems for port 6640 and 6633 on localhost.
I'm using Pike (installed by Packstack). On Queens it happens too, but not so easy. I have 1 Controller and 2 Networks nodes. On Controller, the only problem seems to be the remote neutron-openvswitch-agent down. These logs bellow are from one of the Network nodes. If I restart the neutron-openvswitch-agent on this Network node, normally it works again.

Anybody knows if there a limit for bridges? I changes some timeouts for rpc, ovsdb, increased linux "open files" for openvswitch-agent PID(default 1024/4096), ... Where can I investigate more? Some Idea?

2018-08-30 08:25:29.965 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-736a8da0-fe3d-45dc-839b-4fe020b8208e - - - - -] Switch connection timeout: RuntimeError: ofctl request version=0x4,msg_type=0xe,msg_len=0x40,xid=0x6ff86b1,OFPFlowMod(buffer_id=4294967295,command=3,cookie=3807028968188950860L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'in_port': 2689}),out_group=4294967295,out_port=4294967295,priority=0,table_id=24) timed out
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-736a8da0-fe3d-45dc-839b-4fe020b8208e - - - - -] Error while processing VIF ports: RuntimeError: Switch connection timeout
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2066, in rpc_loop
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ofport_changed_ports = self.update_stale_ofport_rules()
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1210, in update_stale_ofport_rules
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.int_br.delete_arp_spoofing_protection(port=ofport)
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py", line 255, in delete_arp_spoofing_protection
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     match=match)
2018-08-30 08:25:34.110 18858 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2018-09-03 15:05:09 -0600

LuisKleber gravatar image

netstat -an | grep 6633 | grep -v ESTAB

tcp        0      0 127.0.0.1:6633          127.0.0.1:56882         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:54216         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:52774         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:54210         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:51640         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:50708         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:50920         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:51494         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:56522         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:56132         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:52776         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:51638         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:56880         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:54214         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:49874         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:51504         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:56524         SYN_RECV
tcp        0      0 127.0.0.1:6633          127.0.0.1:56130         SYN_RECV
tcp        0      1 127.0.0.1:56834         127.0.0.1:6633          SYN_SENT
tcp        0      0 127.0.0.1:47586         127.0.0.1:6633          TIME_WAIT
tcp        0      0 127.0.0.1:47532         127.0.0.1:6633          TIME_WAIT
tcp        0      1 127.0.0.1:56794         127.0.0.1:6633          SYN_SENT
tcp        0      0 127.0.0.1:43452         127.0.0.1:6633          TIME_WAIT
tcp        0      1 127.0.0.1:56820         127.0.0.1:6633          SYN_SENT
tcp        0      0 127.0.0.1:47610         127.0.0.1:6633          TIME_WAIT
tcp        0      0 127.0.0.1:43598         127.0.0.1:6633          TIME_WAIT
tcp        0      0 127.0.0.1:43460         127.0.0.1:6633          TIME_WAIT
tcp        0      0 127.0.0.1:40884         127.0.0.1:6633          TIME_WAIT
tcp        0      0 127.0.0.1:42042         127.0.0.1:6633          TIME_WAIT
tcp        0      0 127.0.0.1:43656         127.0.0.1:6633          TIME_WAIT
tcp   17      0 127.0.0.1:6633          127.0.0.1:48962         CLOSE_WAIT
tcp        0     17 127.0.0.1:56128         127.0.0.1:6633          FIN_WAIT1
tcp        0      0 127.0.0.1:47576         127.0.0.1:6633          TIME_WAIT
tcp        0     17 127.0.0.1:50708         127.0.0.1:6633          FIN_WAIT1
tcp   17      0 127.0.0.1:6633          127.0.0.1:48922         CLOSE_WAIT
tcp        0      1 ...
(more)
edit flag offensive delete link more

Comments

The problem is with the number of Bridges connected to BR-INT. I reduced to 54 bridges (from 72) and the problems have not appeared anymore.

But, why and what is the problem?

LuisKleber gravatar imageLuisKleber ( 2018-09-18 16:00:12 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2018-09-02 09:49:26 -0600

Seen: 3,985 times

Last updated: Sep 03 '18