Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release.

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release.release. I'm following the installation guide for Ubuntu.

docs.openstack.org/mitaka/install-guide-ubuntu

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release. I'm following the installation guide for Ubuntu.

docs.openstack.org/mitaka/install-guide-ubuntu

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

EDIT: Thanks to @yprokule for helping me troubleshoot this issue. I have made a copy of the adminopenrc script and changed it according to the service credentials sections of the ceilometer conf file. Here's how it looks

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=ceilometer
export OS_PASSWORD=xxxxxxx
export OS_AUTH_URL=http://controller:5000/v2.0
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

I then proceeded to try an issue a token with openstack token issue and here's the output

Could not determine a suitable URL for the plugin

Could be related to the fact that ceilometer client is using v2 when authenticating to keystone?. I'm not sure but from the config files it seems the other services in opesntack are already using v3.

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release. I'm following the installation guide for Ubuntu.

docs.openstack.org/mitaka/install-guide-ubuntu

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

EDIT: EDIT 1: Thanks to @yprokule for helping me troubleshoot this issue. I have made a copy of the adminopenrc script and changed it according to the service credentials sections of the ceilometer conf file. Here's how it looks

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=ceilometer
export OS_PASSWORD=xxxxxxx
export OS_AUTH_URL=http://controller:5000/v2.0
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

I then proceeded to try an issue a token with openstack token issue and here's the output

Could not determine a suitable URL for the plugin

Could be related to the fact that ceilometer client is using v2 when authenticating to keystone?. I'm not sure but from the config files it seems the other services in opesntack are already using v3.

EDIT 2: I also checked the role for user ceilometer in project service and it has an admin role.

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release. I'm following the installation guide for Ubuntu.

docs.openstack.org/mitaka/install-guide-ubuntu

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

EDIT 1: Thanks to @yprokule for helping me troubleshoot this issue. I have made a copy of the adminopenrc script and changed it according to the service credentials sections of the ceilometer conf file. Here's how it looks

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=ceilometer
export OS_PASSWORD=xxxxxxx
export OS_AUTH_URL=http://controller:5000/v2.0
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

I then proceeded to try an issue a token with openstack token issue and here's the output

Could not determine a suitable URL for the plugin

Could be related to the fact that ceilometer client is using v2 when authenticating to keystone?. I'm not sure but from the config files it seems the other services in opesntack openstack are already using v3.

EDIT 2: I also checked the role for user ceilometer in project service and it has an admin role.

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release. I'm following the installation guide for Ubuntu.

docs.openstack.org/mitaka/install-guide-ubuntu

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

EDIT 1: Thanks to @yprokule for helping me troubleshoot this issue. I have made a copy of the adminopenrc script and changed it according to the service credentials sections section of the ceilometer conf file. Here's how it looks

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=ceilometer
export OS_PASSWORD=xxxxxxx
export OS_AUTH_URL=http://controller:5000/v2.0
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

I then proceeded to try an issue a token with openstack token issue and here's the output

Could not determine a suitable URL for the plugin

Could be related to the fact that ceilometer client is using v2 when authenticating to keystone?. I'm not sure but from the config files it seems the other services in openstack are already using v3.

EDIT 2: I also checked the role for user ceilometer in project service and it has an admin role.

Ceilometer - Keystone authentication status code 401

Hi everyone,

At work we've been trying to figure out how to get Ceilometer on the compute node to authenticate agasint Keystone but to no avail.

Here's the output of the keystone-wsgi-public log (controller node)

2016-04-29 18:54:19.074 26606 INFO keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] POST http://controller:5000/v2.0/tokens
2016-04-29 18:54:19.081 26606 WARNING keystone.common.wsgi [req-3e3fab30-5b37-4a22-8ad1-9b42bc46780d - - - - -] Authorization failed. The request you have made requires authentication. from xx.xx.xx.xx
2016-04-29 18:54:19.090 26604 DEBUG keystone.middleware.auth [req-08ae6a8a-4584-4033-9bd0-904d20cb6f1f - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71

And here's the output of the ceilometer-compute.log (compute node)

2016-04-29 18:35:26.708 24510 DEBUG ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:804
2016-04-29 18:35:26.723 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-04-29 18:35:26.724 24510 INFO ceilometer.pipeline [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] detected decoupled pipeline config format
2016-04-29 18:35:26.728 24510 DEBUG keystoneauth.identity.v2 [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Making authentication request to http://controller:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-04-29 18:35:29.135 24510 DEBUG keystoneauth.session [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Request returned failure status: 401 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:466
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     "servers")
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     authenticated=False, log=False)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in request
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, method, url)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-128a4b95-3d5d-486c-bab1-1a9be6f676d1)
2016-04-29 18:35:29.136 24510 ERROR ceilometer.nova_client 
2016-04-29 18:35:29.143 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.144 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.packets, no resources found this cycle
2016-04-29 18:35:29.145 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.iops, no resources found this cycle
2016-04-29 18:35:29.146 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster network.incoming.packets, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-04-29 18:35:29.147 24510 INFO ceilometer.agent.manager [req-34e15155-ebb3-4a29-88ab-c6869c6693c8 admin - - - -] Skip pollster disk.write.requests.rate, no resources found this cycle

This is the keystone auth config on ceilometer.conf

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = CEILOMETER_PASS

And the service credentials

os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS
interface = internalURL
region_name = RegionOne

BTW this is for the Mitaka release. I'm following the installation guide for Ubuntu.

docs.openstack.org/mitaka/install-guide-ubuntu

Please let me know if you need more info. Any pointers would be greatly appreciated

Regards

EDIT 1: Thanks to @yprokule for helping me troubleshoot this issue. I have made a copy of the adminopenrc script and changed it according to the service credentials section of the ceilometer conf file. Here's how it looks

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=ceilometer
export OS_PASSWORD=xxxxxxx
export OS_AUTH_URL=http://controller:5000/v2.0
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

I then proceeded to try an issue a token with openstack token issue and here's the output

Could not determine a suitable URL for the plugin

Could be related to the fact that ceilometer client is using v2 when authenticating to keystone?. I'm not sure but from the config files it seems the other services in openstack are already using v3.

EDIT 2: I also checked the role for user ceilometer in project service and it has an admin role.

EDIT 3: Also the error message in the keystone log makes me suspicious that it requires either a token or certificate to be able to authenticate. I ran a tcpdump between the controller and the compute nodes when restarting the ceilometer-compute agent and this is what I'm seeing:

{"auth": {"tenantName": "service", "passwordCredentials": {"username": "ceilometer", "password": "xxxxxxx"}}}
17:29:55.512941 IP controller.5000 > xx.xx.xx.xx.60256: Flags [.], ack 371, win 235, options [nop,nop,TS val 710719132 ecr 85702763], length 0
E..4ui@.@...

It seems that no matter what other extra parameters I set in ceilomenter.conf the client api is ignoring them. Is this correct?