Revision history [back]

click to hide/show revision 1
initial version

Nova fail to reach Keystone

Hi,

building an architecture with a MySQL cluster as backend. When installing nova (Rocky release), DB are correctly set up with nova-manage api_db/db commands.

From the controller node with admin credentials :

openstack compute service list --debug command

...

Starting new HTTP connection (1): controller:8774

http://controller:8774 "GET /v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services HTTP/1.1" 503 218 RESP: [503] Connection: keep-alive Content-Length: 218 Content-Type: application/json Date: Fri, 23 Aug 2019 07:14:37 GMT X-Compute-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 X-Openstack-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 RESP BODY: {"message": "The server is currently unavailable. Please try again at a later time.

\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} GET call to compute for http://controller:8774/v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services used request id req-1e526faa-2826-4526-9e64-ae6b4dbcc115 The server is currently unavailable. Please try again at a later time.

The Keystone service is temporarily unavailable.

/var/log/nova/nova-api.log

WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure:

Unable to establish connection to http://controller:5000/v3: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.httpconnection object="" at="" 0x707f7fc8bae748="">: Failed to establish a new connection: [Errno -2] No address found'))

2019-08-22 14:42:48.279 2750 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller:5000/v3/auth/tokens: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url

Yet,

  • Nova is able to get a token (token issue command with nova credentials) and access to the MySQL node (and the nova databases)
  • controller is a known host.
  • CLI commands works with glance and neutron services wit the same admin credentials
  • Keystone is running with apache2 and listening on 5000 port

Does anyone had same issues ?

Nova fail to reach Keystone

Hi,

building an architecture with a MySQL cluster as backend. When installing nova (Rocky release), DB are correctly set up with nova-manage api_db/db commands.

From the controller node with admin credentials :

openstack compute service list --debug command

...

Starting new HTTP connection (1): controller:8774

http://controller:8774 "GET /v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services HTTP/1.1" 503 218 RESP: [503] Connection: keep-alive Content-Length: 218 Content-Type: application/json Date: Fri, 23 Aug 2019 07:14:37 GMT X-Compute-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 X-Openstack-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 RESP BODY: {"message": "The server is currently unavailable. Please try again at a later time.

\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} GET call to compute for http://controller:8774/v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services used request id req-1e526faa-2826-4526-9e64-ae6b4dbcc115 The server is currently unavailable. Please try again at a later time.

The Keystone service is temporarily unavailable.

/var/log/nova/nova-api.log

WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure:

Unable to establish connection to http://controller:5000/v3: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.httpconnection object="" at="" 0x707f7fc8bae748="">: Failed to establish a new connection: [Errno -2] No address found'))

2019-08-22 14:42:48.279 2750 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller:5000/v3/auth/tokens: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url

Yet,

  • Nova is able to get a token (token issue command with nova credentials) and access to the MySQL node (and the nova databases)
  • controller is a known host.
  • CLI commands works with glance and neutron services wit the same admin credentials
  • Keystone is running with apache2 and listening on 5000 port

Does anyone had same issues ?

Nova fail fails to reach Keystone

Hi,

building an architecture with a MySQL cluster as backend. When installing nova (Rocky release), DB are correctly set up with nova-manage api_db/db commands.

From the controller node with admin credentials :

openstack compute service list --debug command

...

Starting new HTTP connection (1): controller:8774

http://controller:8774 "GET /v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services HTTP/1.1" 503 218 RESP: [503] Connection: keep-alive Content-Length: 218 Content-Type: application/json Date: Fri, 23 Aug 2019 07:14:37 GMT X-Compute-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 X-Openstack-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 RESP BODY: {"message": "The server is currently unavailable. Please try again at a later time.

\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} GET call to compute for http://controller:8774/v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services used request id req-1e526faa-2826-4526-9e64-ae6b4dbcc115 The server is currently unavailable. Please try again at a later time.

The Keystone service is temporarily unavailable.

/var/log/nova/nova-api.log

WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure:

Unable to establish connection to http://controller:5000/v3: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.httpconnection object="" at="" 0x707f7fc8bae748="">: Failed to establish a new connection: [Errno -2] No address found'))

2019-08-22 14:42:48.279 2750 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller:5000/v3/auth/tokens: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url

Yet,

  • Nova is able to get a token (token issue command with nova credentials) and access to the MySQL node (and the nova databases)
  • controller is a known host.
  • CLI commands works with glance and neutron services wit the same admin credentials
  • Keystone is running with apache2 and listening on 5000 port

Does anyone had same issues ?


Weird but Nova user is unable to parse any other adress than 127.0.0.1 !

Here is my old configuration :

Keystone endpoints for Keystone and Nova services

controller:5000/v3 controller:8774/v2.1

In the nova.conf file, I put auth_url = http:/controller:5000/v3

With error message Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL

The only configuration that makes CLI working is the following one :

127.0.0.1:5000/v3 127.0.0.1:8774/v2.1

In the nova.conf file, I put auth_url = http:/127.0.0.1:5000/v3

Nova fails to reach Keystone

Hi,

building an architecture with a MySQL cluster as backend. When installing nova (Rocky release), DB are correctly set up with nova-manage api_db/db commands.

From the controller node with admin credentials :

openstack compute service list --debug command

...

Starting new HTTP connection (1): controller:8774

http://controller:8774 "GET /v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services HTTP/1.1" 503 218 RESP: [503] Connection: keep-alive Content-Length: 218 Content-Type: application/json Date: Fri, 23 Aug 2019 07:14:37 GMT X-Compute-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 X-Openstack-Request-Id: req-1e526faa-2826-4526-9e64-ae6b4dbcc115 RESP BODY: {"message": "The server is currently unavailable. Please try again at a later time.

\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} GET call to compute for http://controller:8774/v2.1/bb1e75d0099c4df5a79c4a8773cb40dc/os-services used request id req-1e526faa-2826-4526-9e64-ae6b4dbcc115 The server is currently unavailable. Please try again at a later time.

The Keystone service is temporarily unavailable.

/var/log/nova/nova-api.log

WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure:

Unable to establish connection to http://controller:5000/v3: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.httpconnection object="" at="" 0x707f7fc8bae748="">: Failed to establish a new connection: [Errno -2] No address found'))

2019-08-22 14:42:48.279 2750 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller:5000/v3/auth/tokens: HTTPConnectionPool(host='controller', port=5000): Max retries exceeded with url

Yet,

  • Nova is able to get a token (token issue command with nova credentials) and access to the MySQL node (and the nova databases)
  • controller is a known host.
  • CLI commands works with glance and neutron services wit the same admin credentials
  • Keystone is running with apache2 and listening on 5000 port

Does anyone had same issues ?


Weird but Nova user is unable to parse any other adress than 127.0.0.1 !

(or localhost) ! All other services like glance or neutron are able to use controller vip hostname to connect to keystone. What is strange is the fact that actually, nova connect to keystone because it can get a token using controller:5000/v3 address. Here is my old configuration :

Keystone endpoints for Keystone and Nova services

controller:5000/v3 controller:8774/v2.1

In the nova.conf file, I put auth_url = http:/controller:5000/v3

With error message Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL

The only configuration that makes CLI working is the following one :

127.0.0.1:5000/v3 127.0.0.1:8774/v2.1controller:8774/v2.1

In the nova.conf file, I put auth_url = http:/127.0.0.1:5000/v3

May the problem come from the python-novaclient package from debian 10 I use ?