Gnocchi measures data is behind the time

asked 2016-09-08 01:21:14 -0500

Damon gravatar image

Hi,

I am trying Gnocchi with Openstack Liberity and Ubuntu server 14.04. I have successfully run it seprately in a Python virtualenv and integrated it with Liberty.

We are testing it with 100 instances. The problem is that after running for a while(less than 30 min), although Ceilometer is keep sending raw data to Gnocchi-api and gnocchi-metricd is keep processing steadily, the measure data is behind the actual time, the newest measures would be 10 min ago. For a 7 hrs period we only have 5 hrs measures data.

Here is a part of the gnocchi-metricd.log

2016-09-08 14:07:40.984 42512 DEBUG gnocchi.storage._carbonara [-] Computed new metric c972638d-09fd-4769-838f-e5c0c202062f with 1 new measures in 0.59 seconds (827 points/s, 20 measures/s) process_new_measures /var/lib/gnocchi/gnocchi/local/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:408
2016-09-08 14:07:40.986 42512 DEBUG gnocchi.storage [-] Processing new measures process_background_tasks /var/lib/gnocchi/gnocchi/local/lib/python2.7/site-packages/gnocchi/storage/__init__.py:186
2016-09-08 14:07:40.998 42512 DEBUG gnocchi.storage._carbonara [-] Processing measures for 4cf00ac8-9e2d-4b74-9d4e-abb5d6e94a6a process_new_measures /var/lib/gnocchi/gnocchi/local/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:323
2016-09-08 14:07:40.999 42512 DEBUG gnocchi.storage._carbonara [-] Retrieve unaggregated measures for 4cf00ac8-9e2d-4b74-9d4e-abb5d6e94a6a in 0.00s process_new_measures /var/lib/gnocchi/gnocchi/local/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:342
2016-09-08 14:07:41.043 42522 DEBUG gnocchi.storage._carbonara [-] Processing measures for 5347d6ac-f42b-4fe8-ad7c-8fdf6113bd64 process_new_measures /var/lib/gnocchi/gnocchi/local/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:323
2016-09-08 14:07:41.044 42522 DEBUG gnocchi.storage._carbonara [-] Retrieve unaggregated measures for 5347d6ac-f42b-4fe8-ad7c-8fdf6113bd64 in 0.00s process_new_measures /var/lib/gnocchi/gnocchi/local/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:342

I am just wondering if it is because of the amount of raw data (5,816 metrics total). The log of gnocchi-metricd doesn't show any clue about it dumping raw data.

I can provide more system info. I would like know the cause or solution. Also if anyone could tell any method to scale the gnocchi-api and metricd to multiple nodes. Thanks!

edit retag flag offensive close merge delete