Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Instances are not reachable after restarting one of the servers

After restarting one of the servers while doing some physical tests the instances are not reachable, not even with ping. I accessed to one of them via the OpenStack web tool console and realized that they are not receiving any configuration from OpenStack and as a consequence they can not reach the gateway but they can ping each other because are in the same LAN.

This solution of OpenStack was installed with the Ubuntu autopilot guide from the next web page: https://www.ubuntu.com/download/cloud/autopilot

Using juju GUI I realized that the applications that are installed in the server I restarted are the next ones: Neutron-gateway Base-machine Juju-gui Ceph-osd Ceph-radosgw Ceilometer Neutron-api Keystone Rabbitmq-sever

I found there are two services that are not running, neutron-openvswitch-agent and neutron-vpn-agent. Next are the logs.

for neutron-openvswitch-agent:

    2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 322, in _report_state
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     timeout=timeout, retry=retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = self._waiter.wait(msg_id, timeout)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ea723695c0bd46a69a9c0ac7fd095f81
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-02-08 18:23:25.316 137224 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
2017-02-08 18:42:26.679 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n    ectxt.value = e.inner_exc\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 472, in report_state\n    agent_status = self.plugin.create_or_update_agent(context, agent_state)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 385, in create_or_update_agent\n    return self._create_or_update_agent(context, agent)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 352, in _create_or_update_agent\n    context, agent_state[\'agent_type\'], agent_state[\'host\'])\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 307, in _get_agent_by_type_and_host\n    Agent.host == host).one()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one\n    ret = list(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__\n    self.connect()\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect\n    raise exc\n', u'DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'10.232.213.67\' ([Errno 111] ECONNREFUSED)")\n'].
2017-02-08 18:42:26.679 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-02-08 19:10:22.597 137224 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: 2017-02-08T19:10:22Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
2017-02-08 19:10:22.604 137224 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor Interface name,ofport,external_ids --format=json] dies due to the error: 2017-02-08T19:10:22Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)

For neutron-vpn-agent:

2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent [-] Failed reporting state!
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 686, in _report_state
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     True)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     return method(context, 'report_state', **kwargs)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     retry=self.retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     timeout=timeout, retry=retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     retry=retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     result = self._waiter.wait(msg_id, timeout)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     message = self.waiters.get(msg_id, timeout=timeout)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     'to message ID %s' % msg_id)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID af71bd056e4541d695d505dac1e9e478
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent
2017-02-08 18:23:12.054 196479 WARNING oslo.service.loopingcall [-] Function 'neutron_vpnaas.services.vpn.agent.VPNAgent._report_state' run outlasted interval by 30.01 sec
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.232.213.67' ([Errno 111] ECONNREFUSED)")
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n    ectxt.value = e.inner_exc\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 472, in report_state\n    agent_status = self.plugin.create_or_update_agent(context, agent_state)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 385, in create_or_update_agent\n    return self._create_or_update_agent(context, agent)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 352, in _create_or_update_agent\n    context, agent_state[\'agent_type\'], agent_state[\'host\'])\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 307, in _get_agent_by_type_and_host\n    Agent.host == host).one()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one\n    ret = list(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__\n    self.connect()\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect\n    raise exc\n', u'DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'10.232.213.67\' ([Errno 111] ECONNREFUSED)")\n'].
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent

What could be the cause of this problem and how can be fixed?

Instances are not reachable after restarting one of the servers

After restarting one of the servers while doing some physical tests the instances are not reachable, not even with ping. I accessed to one of them via the OpenStack web tool console and realized that they are not receiving any configuration from OpenStack and as a consequence they can not reach the gateway but they can ping each other because are in the same LAN.

This solution of OpenStack was installed with the Ubuntu autopilot guide from the next web page: https://www.ubuntu.com/download/cloud/autopilot

Using juju GUI I realized that the applications that are installed in the server I restarted are the next ones: Neutron-gateway Base-machine Juju-gui Ceph-osd Ceph-radosgw Ceilometer Neutron-api Keystone Rabbitmq-sever

I found there are two services that are not running, neutron-openvswitch-agent and neutron-vpn-agent. Next are the logs.

for neutron-openvswitch-agent:

    2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 322, in _report_state
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     timeout=timeout, retry=retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = self._waiter.wait(msg_id, timeout)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ea723695c0bd46a69a9c0ac7fd095f81
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-02-08 18:23:25.316 137224 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
2017-02-08 18:42:26.679 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n    ectxt.value = e.inner_exc\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 472, in report_state\n    agent_status = self.plugin.create_or_update_agent(context, agent_state)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 385, in create_or_update_agent\n    return self._create_or_update_agent(context, agent)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 352, in _create_or_update_agent\n    context, agent_state[\'agent_type\'], agent_state[\'host\'])\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 307, in _get_agent_by_type_and_host\n    Agent.host == host).one()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one\n    ret = list(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__\n    self.connect()\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect\n    raise exc\n', u'DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'10.232.213.67\' ([Errno 111] ECONNREFUSED)")\n'].
2017-02-08 18:42:26.679 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-02-08 19:10:22.597 137224 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: 2017-02-08T19:10:22Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
2017-02-08 19:10:22.604 137224 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor Interface name,ofport,external_ids --format=json] dies due to the error: 2017-02-08T19:10:22Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)

For neutron-vpn-agent:

2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent [-] Failed reporting state!
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 686, in _report_state
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     True)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     return method(context, 'report_state', **kwargs)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     retry=self.retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     timeout=timeout, retry=retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     retry=retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     result = self._waiter.wait(msg_id, timeout)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     message = self.waiters.get(msg_id, timeout=timeout)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     'to message ID %s' % msg_id)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID af71bd056e4541d695d505dac1e9e478
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent
2017-02-08 18:23:12.054 196479 WARNING oslo.service.loopingcall [-] Function 'neutron_vpnaas.services.vpn.agent.VPNAgent._report_state' run outlasted interval by 30.01 sec
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.232.213.67' ([Errno 111] ECONNREFUSED)")
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n    ectxt.value = e.inner_exc\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 472, in report_state\n    agent_status = self.plugin.create_or_update_agent(context, agent_state)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 385, in create_or_update_agent\n    return self._create_or_update_agent(context, agent)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 352, in _create_or_update_agent\n    context, agent_state[\'agent_type\'], agent_state[\'host\'])\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 307, in _get_agent_by_type_and_host\n    Agent.host == host).one()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one\n    ret = list(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__\n    self.connect()\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect\n    raise exc\n', u'DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'10.232.213.67\' ([Errno 111] ECONNREFUSED)")\n'].
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent

What could be the cause of this problem and how can be fixed?

I found more logs in neutron-server which is in the same physical server but in another container. Next are the logs:

2017-02-13 13:31:27.080 611890 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.232.213.58:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.

...

2017-02-14 06:45:36.292 1770 WARNING neutron.db.agentschedulers_db [req-577e8c58-0c28-4e69-961e-2f9d4c9d1ae2 - - - - -] No DHCP agents available, skipping rescheduling
2017-02-14 06:46:06.048 1770 WARNING neutron.db.agents_db [req-9531e50e-7440-4f16-ba5d-34d623d1bb19 - - - - -] Agent healthcheck: found 6 dead agents out of 9:
                Type       Last heartbeat host
            L3 agent  2017-02-08 19:10:12 curious
  Open vSwitch agent  2017-02-08 19:09:56 curious
          DHCP agent  2017-02-08 19:09:57 curious
  Loadbalancer agent  2017-02-08 19:10:13 curious
      Metering agent  2017-02-08 19:10:11 curious
      Metadata agent  2017-02-08 19:10:04 curious
2017-02-14 06:46:13.301 1770 WARNING neutron.db.agentschedulers_db [req-577e8c58-0c28-4e69-961e-2f9d4c9d1ae2 - - - - -] No DHCP agents available, skipping rescheduling
2017-02-14 06:46:43.038 1770 WARNING neutron.db.agents_db [req-9531e50e-7440-4f16-ba5d-34d623d1bb19 - - - - -] Agent healthcheck: found 6 dead agents out of 9:
                Type       Last heartbeat host
            L3 agent  2017-02-08 19:10:12 curious
  Open vSwitch agent  2017-02-08 19:09:56 curious
          DHCP agent  2017-02-08 19:09:57 curious
  Loadbalancer agent  2017-02-08 19:10:13 curious
      Metering agent  2017-02-08 19:10:11 curious
      Metadata agent  2017-02-08 19:10:04 curious

Instances are not reachable after restarting one of the servers

After restarting one of the servers while doing some physical tests the instances are not reachable, not even with ping. I accessed to one of them via the OpenStack web tool console and realized that they are not receiving any configuration from OpenStack and as a consequence they can not reach the gateway but they can ping each other because are in the same LAN.

This solution of OpenStack was installed with the Ubuntu autopilot guide from the next home web page: https://www.ubuntu.com/download/cloud/autopilotpage.

Using juju GUI I realized that the applications that are installed in the server I restarted are the next ones: Neutron-gateway Base-machine Juju-gui Ceph-osd Ceph-radosgw Ceilometer Neutron-api Keystone Rabbitmq-sever

I found there are two services that are not running, neutron-openvswitch-agent and neutron-vpn-agent. Next are the logs.

for neutron-openvswitch-agent:

    2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 322, in _report_state
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     timeout=timeout, retry=retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = self._waiter.wait(msg_id, timeout)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ea723695c0bd46a69a9c0ac7fd095f81
2017-02-08 18:23:25.312 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-02-08 18:23:25.316 137224 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
2017-02-08 18:42:26.679 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n    ectxt.value = e.inner_exc\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 472, in report_state\n    agent_status = self.plugin.create_or_update_agent(context, agent_state)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 385, in create_or_update_agent\n    return self._create_or_update_agent(context, agent)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 352, in _create_or_update_agent\n    context, agent_state[\'agent_type\'], agent_state[\'host\'])\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 307, in _get_agent_by_type_and_host\n    Agent.host == host).one()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one\n    ret = list(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__\n    self.connect()\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect\n    raise exc\n', u'DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'10.232.213.67\' ([Errno 111] ECONNREFUSED)")\n'].
2017-02-08 18:42:26.679 137224 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-02-08 19:10:22.597 137224 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: 2017-02-08T19:10:22Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
2017-02-08 19:10:22.604 137224 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor Interface name,ofport,external_ids --format=json] dies due to the error: 2017-02-08T19:10:22Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)

For neutron-vpn-agent:

2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent [-] Failed reporting state!
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 686, in _report_state
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     True)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     return method(context, 'report_state', **kwargs)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     retry=self.retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     timeout=timeout, retry=retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     retry=retry)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     result = self._waiter.wait(msg_id, timeout)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     message = self.waiters.get(msg_id, timeout=timeout)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent     'to message ID %s' % msg_id)
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID af71bd056e4541d695d505dac1e9e478
2017-02-08 18:23:12.051 196479 ERROR neutron.agent.l3.agent
2017-02-08 18:23:12.054 196479 WARNING oslo.service.loopingcall [-] Function 'neutron_vpnaas.services.vpn.agent.VPNAgent._report_state' run outlasted interval by 30.01 sec
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.232.213.67' ([Errno 111] ECONNREFUSED)")
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n    ectxt.value = e.inner_exc\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 472, in report_state\n    agent_status = self.plugin.create_or_update_agent(context, agent_state)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 385, in create_or_update_agent\n    return self._create_or_update_agent(context, agent)\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 352, in _create_or_update_agent\n    context, agent_state[\'agent_type\'], agent_state[\'host\'])\n', u'  File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 307, in _get_agent_by_type_and_host\n    Agent.host == host).one()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one\n    ret = list(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 905, in connection\n    execution_options=execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind\n    engine, execution_options)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind\n    conn = bind.contextual_connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect\n    self._wrap_pool_connect(self.pool.connect, None),\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect\n    e, dialect, self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect\n    return fn()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect\n    return _ConnectionFairy._checkout(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout\n    fairy = _ConnectionRecord.checkout(pool)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout\n    rec.checkin()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout\n    dbapi_connection = rec.get_connection()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection\n    self.connection = self.__connect()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect\n    connection = self.__pool._invoke_creator(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect\n    return dialect.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect\n    return self.dbapi.connect(*cargs, **cparams)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect\n    return Connection(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__\n    self.connect()\n', u'  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect\n    raise exc\n', u'DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'10.232.213.67\' ([Errno 111] ECONNREFUSED)")\n'].
2017-02-08 18:42:12.239 196479 ERROR neutron.agent.l3.agent

What could be the cause of this problem and how can be fixed?

I found more logs in neutron-server which is in the same physical server but in another container. Next are the logs:

2017-02-13 13:31:27.080 611890 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.232.213.58:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.

...

2017-02-14 06:45:36.292 1770 WARNING neutron.db.agentschedulers_db [req-577e8c58-0c28-4e69-961e-2f9d4c9d1ae2 - - - - -] No DHCP agents available, skipping rescheduling
2017-02-14 06:46:06.048 1770 WARNING neutron.db.agents_db [req-9531e50e-7440-4f16-ba5d-34d623d1bb19 - - - - -] Agent healthcheck: found 6 dead agents out of 9:
                Type       Last heartbeat host
            L3 agent  2017-02-08 19:10:12 curious
  Open vSwitch agent  2017-02-08 19:09:56 curious
          DHCP agent  2017-02-08 19:09:57 curious
  Loadbalancer agent  2017-02-08 19:10:13 curious
      Metering agent  2017-02-08 19:10:11 curious
      Metadata agent  2017-02-08 19:10:04 curious
2017-02-14 06:46:13.301 1770 WARNING neutron.db.agentschedulers_db [req-577e8c58-0c28-4e69-961e-2f9d4c9d1ae2 - - - - -] No DHCP agents available, skipping rescheduling
2017-02-14 06:46:43.038 1770 WARNING neutron.db.agents_db [req-9531e50e-7440-4f16-ba5d-34d623d1bb19 - - - - -] Agent healthcheck: found 6 dead agents out of 9:
                Type       Last heartbeat host
            L3 agent  2017-02-08 19:10:12 curious
  Open vSwitch agent  2017-02-08 19:09:56 curious
          DHCP agent  2017-02-08 19:09:57 curious
  Loadbalancer agent  2017-02-08 19:10:13 curious
      Metering agent  2017-02-08 19:10:11 curious
      Metadata agent  2017-02-08 19:10:04 curious