RemoteError: Remote error: TimeoutError QueuePool limit of size x overflow xy reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)

asked 2019-10-23 22:37:24 -0600

Steve987654321 gravatar image

We have an issue with metadata agent. One compute node hosts many vms and the vms can't get ip. I've tried to restart on the dhcp metadata server both service, tried to restart our rabbitmq server as well. Also restarted neutron server, but the issue still persist and the vms are down.

Errors like these:

2019-10-24 10:34:47.794 86154 ERROR neutron.agent.metadata.agent
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [-] Unexpected error.: RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 89, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     ports = self._get_ports(remote_address, network_id, router_id)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self._get_ports_for_remote_address(remote_address, networks)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 116, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self.func(target_self, *args, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     ip_address=remote_address)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self.plugin_rpc.get_ports(self.context, filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 72, in get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return cctxt.call(context, 'get_ports', filters=filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self._original_context.call(ctxt, method, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     retry ...
(more)
edit retag flag offensive close merge delete