Nova fails to reach Keystone
Hi,
building an architecture with a MySQL cluster as backend. When installing nova (Rocky release), DB are correctly set up with nova-manage api_db/db commands.
From the controller node with admin credentials :
openstack compute service list --debug command
...
Starting new HTTP connection (1): controller:8774
http://controller:8774 "GET /v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services HTTP/1.1" 503 218
RESP: [503] Connection: keep-alive Content-Length: 218 Content-Type: application/json Date: Fri, 23 Aug 2019 07:14:37 GMT X-Compute-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 X-Openstack-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115
RESP BODY: {"message": "The server is currently unavailable. Please try again at a later time.
\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"}
GET call to compute for http://controller:8774/v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services (http://controller:8774/v2.1/bb1e75d00...) used request id req-1e526faa-2826-4526-9e64-ae6b4dbcc115
The server is currently unavailable. Please try again at a later time.
The Keystone service is temporarily unavailable.
/var/log/nova/nova-api.log
WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure:
Unable to establish connection to http://controller:5000/v3: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.httpconnection 0x707f7fc8bae748="" at="" object="">: Failed to establish a new connection: [Errno -2] No address found'))</urllib3.connection.httpconnection>
2019-08-22 14:42:48.279 2750 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller:5000/v3/auth/tokens: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url
Yet,
- Nova is able to get a token (token issue command with nova credentials) and access to the MySQL node (and the nova databases)
- controller is a known host.
- CLI commands works with glance and neutron services wit the same admin credentials
- Keystone is running with apache2 and listening on 5000 port
Does anyone had same issues ?
Weird but Nova user is unable to parse any other adress than 127.0.0.1 (or localhost) ! All other services like glance or neutron are able to use controller vip hostname to connect to keystone. What is strange is the fact that actually, nova connect to keystone because it can get a token using controller:5000/v3 address. Here is my old configuration :
Keystone endpoints for Keystone and Nova services
controller:5000/v3 controller:8774/v2.1
In the nova.conf file, I put auth_url = http:/controller:5000/v3
With error message Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL
The only configuration that makes CLI working is the following one :
127.0.0.1:5000/v3 controller:8774/v2.1
In the nova.conf file, I put auth_url = http:/127.0.0.1:5000/v3
May the problem come from the python-novaclient package from debian 10 I use ?
It clearly says "Unable to establish connection to http://controller:5000/v3". Perhaps you misconfigured the Keystone endpoint in nova.conf? Is http://controller:5000/v3 the authentication URL you use when you run openstack token issue?
I am not sure how to interpret "no address found", though.
Similar problem here: https://ask.openstack.org/en/question.... In this case, there was no name resolution for controller. What happens when you run curl http://controller:5000/v3?
controller is a VIP used on the management interface of a load balancer (controller1 and controller2 are two controllers in an active/active state for openstack services). curl http://controller:5000/v3 returns a stable status
Where do you run curl? On the same servers where Nova API is running? If so, this is weird. Do the nova-api processes run in an environment that doesn't give them access to controller? E.g. a container with ill-configured name resolution or hosts file?
curl is run on the controller1 and controller2 nodes. Pacemaker is used to attribute the vip "controller" to one of the nodes. Nova API service is running on both. The /etc/hosts file is configured to all the nodes to translate the IP address to the name "controller". Same problems on compute node