Ask Your Question
0

Nova fails to reach Keystone

asked 2019-08-23 02:58:56 -0500

neua gravatar image

updated 2019-08-26 05:16:24 -0500

Hi,

building an architecture with a MySQL cluster as backend. When installing nova (Rocky release), DB are correctly set up with nova-manage api_db/db commands.

From the controller node with admin credentials :

openstack compute service list --debug command

...

Starting new HTTP connection (1): controller:8774

http://controller:8774 "GET /v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services HTTP/1.1" 503 218 RESP: [503] Connection: keep-alive Content-Length: 218 Content-Type: application/json Date: Fri, 23 Aug 2019 07:14:37 GMT X-Compute-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 X-Openstack-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 RESP BODY: {"message": "The server is currently unavailable. Please try again at a later time.

\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} GET call to compute for http://controller:8774/v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services (http://controller:8774/v2.1/bb1e75d00...) used request id req-1e526faa-2826-4526-9e64-ae6b4dbcc115 The server is currently unavailable. Please try again at a later time.

The Keystone service is temporarily unavailable.

/var/log/nova/nova-api.log

WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure:

Unable to establish connection to http://controller:5000/v3: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.httpconnection 0x707f7fc8bae748="" at="" object="">: Failed to establish a new connection: [Errno -2] No address found'))</urllib3.connection.httpconnection>

2019-08-22 14:42:48.279 2750 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller:5000/v3/auth/tokens: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url

Yet,

  • Nova is able to get a token (token issue command with nova credentials) and access to the MySQL node (and the nova databases)
  • controller is a known host.
  • CLI commands works with glance and neutron services wit the same admin credentials
  • Keystone is running with apache2 and listening on 5000 port

Does anyone had same issues ?


Weird but Nova user is unable to parse any other adress than 127.0.0.1 (or localhost) ! All other services like glance or neutron are able to use controller vip hostname to connect to keystone. What is strange is the fact that actually, nova connect to keystone because it can get a token using controller:5000/v3 address. Here is my old configuration :

Keystone endpoints for Keystone and Nova services

controller:5000/v3 controller:8774/v2.1

In the nova.conf file, I put auth_url = http:/controller:5000/v3

With error message Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL

The only configuration that makes CLI working is the following one :

127.0.0.1:5000/v3 controller:8774/v2.1

In the nova.conf file, I put auth_url = http:/127.0.0.1:5000/v3

May the problem come from the python-novaclient package from debian 10 I use ?

edit retag flag offensive close merge delete

Comments

It clearly says "Unable to establish connection to http://controller:5000/v3". Perhaps you misconfigured the Keystone endpoint in nova.conf? Is http://controller:5000/v3 the authentication URL you use when you run openstack token issue?

I am not sure how to interpret "no address found", though.

Bernd Bausch gravatar imageBernd Bausch ( 2019-08-23 05:52:51 -0500 )edit

Similar problem here: https://ask.openstack.org/en/question.... In this case, there was no name resolution for controller. What happens when you run curl http://controller:5000/v3?

Bernd Bausch gravatar imageBernd Bausch ( 2019-08-23 05:54:46 -0500 )edit

controller is a VIP used on the management interface of a load balancer (controller1 and controller2 are two controllers in an active/active state for openstack services). curl http://controller:5000/v3 returns a stable status

neua gravatar imageneua ( 2019-08-23 06:20:59 -0500 )edit

Where do you run curl? On the same servers where Nova API is running? If so, this is weird. Do the nova-api processes run in an environment that doesn't give them access to controller? E.g. a container with ill-configured name resolution or hosts file?

Bernd Bausch gravatar imageBernd Bausch ( 2019-08-26 02:07:20 -0500 )edit

curl is run on the controller1 and controller2 nodes. Pacemaker is used to attribute the vip "controller" to one of the nodes. Nova API service is running on both. The /etc/hosts file is configured to all the nodes to translate the IP address to the name "controller". Same problems on compute node

neua gravatar imageneua ( 2019-08-26 02:32:35 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2019-08-27 06:48:13 -0500

neua gravatar image

Problem resolved by simply replace all the hostname "controller" by the IP address associated. Nova, Cinder and Placement users don't seem to translate as well hostnames as their partners.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-08-23 02:58:09 -0500

Seen: 96 times

Last updated: Aug 27