Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

How to fix Ceilometer SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I have a test instance of Mirantis OpenStack setup where I have all the services listening only on the managment network and HAproxy is setup with HTTPS for the public endpoints. So my endpoint list looks like

+---------------------------------------------------+--------------------------------------------------+-----------------------------------------------+
|                    publicurl                      |                  internalurl                     |                  adminurl                     |
+---------------------------------------------------+--------------------------------------------------+-----------------------------------------------+
|    https://PBC.EXT.IP.ADR:8773/services/Cloud     |     http://MGM.INT.IP.ADR:8773/services/Cloud    |   http://MGM.INT.IP.ADR:8773/services/Admin   |
|            https://PBC.EXT.IP.ADR:8777            |            http://MGM.INT.IP.ADR:8777            |          http://MGM.INT.IP.ADR:8777           |
|           https://PBC.EXT.IP.ADR:9696/            |            http://MGM.INT.IP.ADR:9696/           |          http://MGM.INT.IP.ADR:9696/          |
|          https://PBC.EXT.IP.ADR:8000/v1/          |          http://MGM.INT.IP.ADR:8000/v1/          |        http://MGM.INT.IP.ADR:8000/v1/         |
|  https://PBC.EXT.IP.ADR:8386/v1.1/%(tenant_id)s   |   http://MGM.INT.IP.ADR:8386/v1.1/%(tenant_id)s  | http://MGM.INT.IP.ADR:8386/v1.1/%(tenant_id)s |
|   https://PBC.EXT.IP.ADR:8082/v1/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8082/v1/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8082/v1/%(tenant_id)s  |
|   https://PBC.EXT.IP.ADR:8774/v2/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8774/v2/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8774/v2/%(tenant_id)s  |
|            https://PBC.EXT.IP.ADR:9292            |            http://MGM.INT.IP.ADR:9292            |          http://MGM.INT.IP.ADR:9292           |
|   https://PBC.EXT.IP.ADR:8776/v2/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8776/v2/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8776/v2/%(tenant_id)s  |
|   https://PBC.EXT.IP.ADR:8776/v1/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8776/v1/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8776/v1/%(tenant_id)s  |
|         https://PBC.EXT.IP.ADR:5000/v2.0          |          http://MGM.INT.IP.ADR:5000/v2.0         |       http://MGM.INT.IP.ADR:35357/v2.0        |
|   https://PBC.EXT.IP.ADR:8004/v1/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8004/v1/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8004/v1/%(tenant_id)s  |
| https://PBC.EXT.IP.ADR:8080/v1/AUTH_%(tenant_id)s | http://MGM.INT.IP.ADR:8080/v1/AUTH_%(tenant_id)s |          http://MGM.INT.IP.ADR:8080/          |
|            https://PBC.EXT.IP.ADR:8080            |            http://MGM.INT.IP.ADR:8080            |          http://MGM.INT.IP.ADR:8080           |
+---------------------------------------------------+--------------------------------------------------+-----------------------------------------------+

Now I'm using a self signed certificate and it looks like ceilometer doesn't have a way to tell it not to verify the certificate. Here is what I see in the logs (I'm guessing it is trying to use the public URL to talk to nova)

<131>May 18 20:16:04 node-51 ceilometer-agent-central Unable to discover resources: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent Traceback (most recent call last):
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/agent.py", line 233, in discover
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     discovered = discoverer.discover(self, param)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/hardware/discovery.py", line 51, in discover
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     instances = self.nova_cli.instance_get_all()
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/nova_client.py", line 50, in with_logging
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return func(*args, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/nova_client.py", line 153, in instance_get_all
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     search_opts=search_opts)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 603, in list
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return self._list("/servers%s%s" % (detail, query_string), "servers")
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 67, in _list
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     _resp, body = self.api.client.get(url)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 487, in get
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return self._cs_request(url, 'GET', **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 465, in _cs_request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     resp, body = self._time_request(url, method, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 439, in _time_request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     resp, body = self.request(url, method, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 410, in request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return session.request(method=method, url=url, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     resp = self.send(prep, **send_kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     r = adapter.send(request, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 331, in send
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     raise SSLError(e)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent

and here is what my /etc/ceilometer/ceilometer.conf looks like

[DEFAULT]
rabbit_hosts=MGM.INT.IP.ADR:5673,MGM.INT.IP.ADR:5673
rabbit_use_ssl=False
rabbit_userid=nova
rabbit_password=SECRECT
rabbit_virtual_host=/
rabbit_ha_queues=True
notification_topics=notifications
rpc_backend=ceilometer.openstack.common.rpc.impl_kombu
debug=False
verbose=True
log_dir=/var/log/ceilometer
use_syslog=True
use_syslog_rfc_format=True
syslog_log_facility=LOG_LOCAL0
[alarm]
evaluation_service=ceilometer.alarm.service.SingletonAlarmService
rest_notifier_ssl_verify=false
partition_rpc_topic=alarm_partition_coordination
evaluation_interval=60
record_history=True
[api]
port=8777
host=MGM.INT.IP.ADR
[collector]
[database]
connection=mongodb://ceilometer:SECRET@MGM.INT.IP.ADR,MGM.INT.IP.ADR,MGM.INT.IP.ADR/ceilometer
[dispatcher_file]
[event]
[keystone_authtoken]
auth_host=MGM.INT.IP.ADR
auth_port=35357
auth_protocol=http
auth_uri=http://MGM.INT.IP.ADR:5000/
admin_user=ceilometer
admin_password=SECRET
admin_tenant_name=services
cafile=/etc/pki/tls/certs/ca.pem
insecure=true
[matchmaker_ring]
[notification]
ack_on_event_error=True
store_events=False
[publisher]
metering_secret=SECRET
[publisher_rpc]
[service_credentials]
os_username=ceilometer
os_password=SECRET
os_tenant_name=services
os_auth_url=http://MGM.INT.IP.ADR:5000/v2.0
os_region_name=regionOne
[vmware]

any and all help is very appreciated.

How to fix Ceilometer SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I have a test instance of Mirantis OpenStack setup where I have all the services listening only on the managment network and HAproxy is setup with HTTPS for the public endpoints. So my endpoint list looks like

+---------------------------------------------------+--------------------------------------------------+-----------------------------------------------+
|                    publicurl                      |                  internalurl                     |                  adminurl                     |
+---------------------------------------------------+--------------------------------------------------+-----------------------------------------------+
|    https://PBC.EXT.IP.ADR:8773/services/Cloud     |     http://MGM.INT.IP.ADR:8773/services/Cloud    |   http://MGM.INT.IP.ADR:8773/services/Admin   |
|            https://PBC.EXT.IP.ADR:8777            |            http://MGM.INT.IP.ADR:8777            |          http://MGM.INT.IP.ADR:8777           |
|           https://PBC.EXT.IP.ADR:9696/            |            http://MGM.INT.IP.ADR:9696/           |          http://MGM.INT.IP.ADR:9696/          |
|          https://PBC.EXT.IP.ADR:8000/v1/          |          http://MGM.INT.IP.ADR:8000/v1/          |        http://MGM.INT.IP.ADR:8000/v1/         |
|  https://PBC.EXT.IP.ADR:8386/v1.1/%(tenant_id)s   |   http://MGM.INT.IP.ADR:8386/v1.1/%(tenant_id)s  | http://MGM.INT.IP.ADR:8386/v1.1/%(tenant_id)s |
|   https://PBC.EXT.IP.ADR:8082/v1/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8082/v1/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8082/v1/%(tenant_id)s  |
|   https://PBC.EXT.IP.ADR:8774/v2/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8774/v2/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8774/v2/%(tenant_id)s  |
|            https://PBC.EXT.IP.ADR:9292            |            http://MGM.INT.IP.ADR:9292            |          http://MGM.INT.IP.ADR:9292           |
|   https://PBC.EXT.IP.ADR:8776/v2/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8776/v2/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8776/v2/%(tenant_id)s  |
|   https://PBC.EXT.IP.ADR:8776/v1/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8776/v1/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8776/v1/%(tenant_id)s  |
|         https://PBC.EXT.IP.ADR:5000/v2.0          |          http://MGM.INT.IP.ADR:5000/v2.0         |       http://MGM.INT.IP.ADR:35357/v2.0        |
|   https://PBC.EXT.IP.ADR:8004/v1/%(tenant_id)s    |    http://MGM.INT.IP.ADR:8004/v1/%(tenant_id)s   |  http://MGM.INT.IP.ADR:8004/v1/%(tenant_id)s  |
| https://PBC.EXT.IP.ADR:8080/v1/AUTH_%(tenant_id)s | http://MGM.INT.IP.ADR:8080/v1/AUTH_%(tenant_id)s |          http://MGM.INT.IP.ADR:8080/          |
|            https://PBC.EXT.IP.ADR:8080            |            http://MGM.INT.IP.ADR:8080            |          http://MGM.INT.IP.ADR:8080           |
+---------------------------------------------------+--------------------------------------------------+-----------------------------------------------+

Now I'm using a self signed certificate and it looks like ceilometer doesn't have a way to tell it not to verify the certificate. Here is what I see in the logs (I'm guessing it is trying to use the public URL to talk to nova)

<131>May 18 20:16:04 node-51 ceilometer-agent-central Unable to discover resources: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent Traceback (most recent call last):
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/agent.py", line 233, in discover
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     discovered = discoverer.discover(self, param)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/hardware/discovery.py", line 51, in discover
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     instances = self.nova_cli.instance_get_all()
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/nova_client.py", line 50, in with_logging
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return func(*args, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/ceilometer/nova_client.py", line 153, in instance_get_all
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     search_opts=search_opts)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 603, in list
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return self._list("/servers%s%s" % (detail, query_string), "servers")
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 67, in _list
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     _resp, body = self.api.client.get(url)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 487, in get
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return self._cs_request(url, 'GET', **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 465, in _cs_request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     resp, body = self._time_request(url, method, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 439, in _time_request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     resp, body = self.request(url, method, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 410, in request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     return session.request(method=method, url=url, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     resp = self.send(prep, **send_kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     r = adapter.send(request, **kwargs)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent   File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 331, in send
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent     raise SSLError(e)
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-05-18 20:16:04.889 31763 TRACE ceilometer.agent

and here is what my /etc/ceilometer/ceilometer.conf looks like

[DEFAULT]
rabbit_hosts=MGM.INT.IP.ADR:5673,MGM.INT.IP.ADR:5673
rabbit_use_ssl=False
rabbit_userid=nova
rabbit_password=SECRECT
rabbit_virtual_host=/
rabbit_ha_queues=True
notification_topics=notifications
rpc_backend=ceilometer.openstack.common.rpc.impl_kombu
debug=False
verbose=True
log_dir=/var/log/ceilometer
use_syslog=True
use_syslog_rfc_format=True
syslog_log_facility=LOG_LOCAL0
[alarm]
evaluation_service=ceilometer.alarm.service.SingletonAlarmService
rest_notifier_ssl_verify=false
partition_rpc_topic=alarm_partition_coordination
evaluation_interval=60
record_history=True
[api]
port=8777
host=MGM.INT.IP.ADR
[collector]
[database]
connection=mongodb://ceilometer:SECRET@MGM.INT.IP.ADR,MGM.INT.IP.ADR,MGM.INT.IP.ADR/ceilometer
[dispatcher_file]
[event]
[keystone_authtoken]
auth_host=MGM.INT.IP.ADR
auth_port=35357
auth_protocol=http
auth_uri=http://MGM.INT.IP.ADR:5000/
admin_user=ceilometer
admin_password=SECRET
admin_tenant_name=services
cafile=/etc/pki/tls/certs/ca.pem
insecure=true
[matchmaker_ring]
[notification]
ack_on_event_error=True
store_events=False
[publisher]
metering_secret=SECRET
[publisher_rpc]
[service_credentials]
os_username=ceilometer
os_password=SECRET
os_tenant_name=services
os_auth_url=http://MGM.INT.IP.ADR:5000/v2.0
os_region_name=regionOne
[vmware]

any and all help is very appreciated.