RemoteError: Remote error: TimeoutError QueuePool limit of size x overflow xy reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
We have an issue with metadata agent. One compute node hosts many vms and the vms can't get ip. I've tried to restart on the dhcp metadata server both service, tried to restart our rabbitmq server as well. Also restarted neutron server, but the issue still persist and the vms are down.
Errors like these:
2019-10-24 10:34:47.794 86154 ERROR neutron.agent.metadata.agent
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [-] Unexpected error.: RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 89, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 116, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.func(target_self, *args, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ip_address=remote_address)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 72, in get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry ...