Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Ceilometer meter-list throws an 500 error

I ran out of disk space, thanks to resource hungry mongodb. To clear the space, i deleted all the contents under the data directory of mongodb "/var/lib/mongodb" in my case.

Now, after restarting the ceilometer services, all my ceilometer CLI commands are giving a 500 error.

Upon checking the logs, here is what i found.

2017-03-06 17:22:37.600 31112 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-pac
kages/ceilometer/storage/__init__.py:99
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks [-] Failed to connect to db, purpose metering retry later: command SON([('authenticate', 1), ('user', u'ce
ilometer'), ('nonce', u'3d7e46ae1fc574d2'), ('key', u'39d8c9a497a313e5a8801bea33785ef8')]) on namespace ceilometer.$cmd failed: auth failed
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks Traceback (most recent call last):
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/api/hooks.py", line 62, in get_connection
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return storage.get_connection_from_config(cfg.CONF, purpose)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 88, in get_connection_from_
config
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return _inner()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return _inner()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     raise attempt.get()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     six.reraise(self.value[0], self.value[1], self.value[2])
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 86, in _inner
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return get_connection(url, namespace)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 101, in get_connection
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return mgr.driver(url)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/storage/impl_mongodb.py", line 148, in __init__
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     self.version = self.conn.server_info()['versionArray']
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/storage/mongo/utils.py", line 406, in closure
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return call(*args, **kwargs)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib/python2.7/site-packages/ceilometer/storage/mongo/utils.py", line 428, in __call__
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return self.method(*args, **kwargs)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 898, in server_info
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     read_preference=ReadPreference.PRIMARY)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 451, in command
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return self.gen.next()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 699, in _socket_for_reads
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     with self._get_socket(read_preference) as sock_info:
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return self.gen.next()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 665, in _get_socket
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     with server.get_socket(self.__all_credentials) as sock_info:
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return self.gen.next()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 102, in get_socket
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     with self.pool.get_socket(all_credentials, checkout) as sock_info:
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return self.gen.next()
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 511, in get_socket
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     sock_info.check_auth(all_credentials)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 276, in check_auth
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     auth.authenticate(credentials, self)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 407, in authenticate
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     auth_func(credentials, sock_info)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 389, in _authenticate_default
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     return _authenticate_mongo_cr(credentials, sock_info)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 382, in _authenticate_mongo_cr
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     sock_info.command(source, query)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 184, in command
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     codec_options, check, allowable_errors)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 54, in command
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     helpers._check_command_response(response_doc, msg, allowable_errors)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 188, in _check_command_response
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks     raise OperationFailure(msg % errmsg, code, response)
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks OperationFailure: command SON([('authenticate', 1), ('user', u'ceilometer'), ('nonce', u'3d7e46ae1fc574d2'), ('key', u'39d8c9a497a313e5a8801bea33785ef8')]) on namespace ceilometer.$cmd failed: auth failed
2017-03-06 17:22:37.603 31112 ERROR ceilometer.api.hooks
2017-03-06 17:22:37.609 31112 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packag

I noticed that ceilometer.ns file in absent in the data_dir for mongodb. and perhaps that is the error.

can someone guide me on this. PS: My configuration file are all OK, because the setup was working before as expected.

Have i d