Ask Your Question

Steve987654321's profile - activity

2019-10-23 22:37:24 -0500 asked a question RemoteError: Remote error: TimeoutError QueuePool limit of size x overflow xy reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)

We have an issue with metadata agent. One compute node hosts many vms and the vms can't get ip. I've tried to restart on the dhcp metadata server both service, tried to restart our rabbitmq server as well. Also restarted neutron server, but the issue still persist and the vms are down.

Errors like these:

2019-10-24 10:34:47.794 86154 ERROR neutron.agent.metadata.agent
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [-] Unexpected error.: RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 89, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     ports = self._get_ports(remote_address, network_id, router_id)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self._get_ports_for_remote_address(remote_address, networks)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 116, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self.func(target_self, *args, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     ip_address=remote_address)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self.plugin_rpc.get_ports(self.context, filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 72, in get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return cctxt.call(context, 'get_ports', filters=filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     return self._original_context.call(ctxt, method, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent     retry ...
(more)
2019-01-13 10:14:10 -0500 received badge  Famous Question (source)
2018-10-20 12:08:07 -0500 received badge  Popular Question (source)
2018-10-20 12:08:07 -0500 received badge  Notable Question (source)
2018-09-03 13:55:31 -0500 asked a question Set grafana collectd need to make visibility of all of our clusters

Hi,

Let me define first what is my goal:
I want to have pretty Grafana dashboards about our openstack clusters. We have 5 datacenters with around 3-4000 physical machine and 15k vm-s. My task is to create some pretty Grafana dashboards fo mysql things, rabbit mq things and of course about the openstack performance, instance counts … We want to use it for capacity planning os something like that, for example in the last 1 year how many xxlarge vm-s were requested in each tenant.

The last week I read a lot about this because I completely new in this area and now I completely got lost regarding the objects, gnocchi, ceilometer, telemetry, aodh, collectd, Prometheus, influxdb … so in my head at the moment I have a completely big mess.

Our current Grafana infrastructure is actually built, I should integrate our openstack stuffs into it.

  1. Collectd is running everywhere
  2. Graphite is collecting the collect things
  3. Grafana is reading the graphite

I can modify, add more source to the graphite server, but please help me, what should I do, which is the best was to make it work?

I’ve tried dashboards imported from Grafana but I can’t make it work, it always show idiot information.

So how is should collect the data? Gnocchi or collectd or something else?

If I collect with gnocchi do I have to add new source for graphite where we have the data?

Do I need aodh or ceilometer (telemetry)?

Our DC-s looks like the following:

  • 2 Controller nodes running neutron, nova, keystone
  • Separated load balancers
  • separated rabbit mq
  • separated db servers