Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Neutron agents dbconectionerror

Hi All,

I am running an OpenStack deployment with mutiple instances of each service behind haproxy. The database connections are to a galera cluster though the same set of load balancers.

I am having some issues with Neutron agents displaying the following error in their logs:

    2017-03-15 14:23:28.398 22029 ERROR neutron Traceback (most recent call last):
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2017-03-15 14:23:28.398 22029 ERROR neutron     sys.exit(main())
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2017-03-15 14:23:28.398 22029 ERROR neutron     agent_main.main()
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2017-03-15 14:23:28.398 22029 ERROR neutron     mod.main()
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main
2017-03-15 14:23:28.398 22029 ERROR neutron     ovs_neutron_agent.main(bridge_classes)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2160, in main
2017-03-15 14:23:28.398 22029 ERROR neutron     agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 237, in __init__
2017-03-15 14:23:28.398 22029 ERROR neutron     self.dvr_agent.setup_dvr_flows()
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 149, in setup_dvr_flows
2017-03-15 14:23:28.398 22029 ERROR neutron     self.setup_dvr_mac_flows_on_all_brs()
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 318, in setup_dvr_mac_flows_on_all_brs
2017-03-15 14:23:28.398 22029 ERROR neutron     dvr_macs = self.plugin_rpc.get_dvr_mac_address_list(self.context)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 46, in wrapper
2017-03-15 14:23:28.398 22029 ERROR neutron     return method(*args, **kwargs)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/dvr_rpc.py", line 52, in get_dvr_mac_address_list
2017-03-15 14:23:28.398 22029 ERROR neutron     return cctxt.call(context, 'get_dvr_mac_address_list')
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 136, in call
2017-03-15 14:23:28.398 22029 ERROR neutron     return self._original_context.call(ctxt, method, **kwargs)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-03-15 14:23:28.398 22029 ERROR neutron     retry=self.retry)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2017-03-15 14:23:28.398 22029 ERROR neutron     timeout=timeout, retry=retry)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-03-15 14:23:28.398 22029 ERROR neutron     retry=retry)
2017-03-15 14:23:28.398 22029 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2017-03-15 14:23:28.398 22029 ERROR neutron     raise result
2017-03-15 14:23:28.398 22029 ERROR neutron RemoteError: Remote error: DBConnectionError (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"')
2017-03-15 14:23:28.398 22029 ERROR neutron [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/dvr_rpc.py", line 92, in get_dvr_mac_address_list\n    return self.plugin.get_dvr_mac_address_list(context)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/db/dvr_mac_db.py", line 144, in get_dvr_mac_address_list\n    query(DistributedVirtualRouterMacAddress).all())\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all\n    return list(self)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__\n', u'DBConnectionError: (_mysql_exceptions.OperationalError) (2013, \'Lost connection to MySQL server at \\\'reading initial communication packet\\\', system error: 0 "Internal error/check (Not system error)"\')\n'].
2017-03-15 14:23:28.398 22029 ERROR neutron
2017-03-15 14:23:28.409 22029 INFO oslo_rootwrap.client [req-ad63cb1e-cbaa-415b-9a89-0845707d55e7 - - - - -] Stopping rootwrap daemon process with pid=22060

Creating of Instances works although no dhcp agent is responding. Api calls (neutron agent-list) seem to be working fine and the only errors in the neutron server log seem to point to being unable to communicate with the agents:

2017-03-15 14:31:50.866 31246 ERROR neutron.plugins.ml2.managers [req-282dfe26-c996-440e-98e3-22f85d2025fd - - - - -] Failed to bind port 43dcc189-eb9a-4cfd-bb47-7080c2cef0dd on host vm165.nubes.stfc.ac.uk for vnic_type normal using segments [{'segmentation_id': 100L, 'physical_network': None, 'id': u'f11a6e44-12a6-4f2d-88f9-d8be649d4137', 'network_type': u'vxlan'}]

Thanks

Alex