Ask Your Question
0

openstack compute service list: Unknown Error (HTTP 503) / Service Unavailable

asked 2016-06-21 08:18:11 -0600

pascalnok gravatar image

Hi all,

I try to install a controller (following http://docs.openstack.org/mitaka/install-guide-rdo). All is OK until the verification of services components:

root@angelus ~ # openstack compute service list
Unknown Error (HTTP 503) (Request-ID: req-64e7d1fb-63b6-42d5-87f3-414580caf15d)

With --debug option, the error "503 Service Unavailable" is additionally displayed:

REQ: curl -g -i -X GET http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d/os-services ....
Starting new HTTP connection (1): controller
"GET /v2.1/4d5fc54a120c42b9b76ff438c944db0d/os-services HTTP/1.1" 503 100
RESP: [503] Date: Tue, 21 Jun 2016 12:05:37 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 100 X-Compute-Request-Id: req-a6a5e831-c3c4-47f7-ba49-484a506c5340
RESP BODY: 503 Service Unavailable
The server is currently unavailable. Please try again at a later time.

And I can see the status of 2 services (nova-osapi_compute and nova-metadata) returned by nova-manage service list is "XXX":

Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth angelus                              internal         enabled    :-)   2016-06-21 12:29:46
nova-conductor   angelus                              internal         enabled    :-)   2016-06-21 12:29:51
nova-scheduler   angelus                              internal         enabled    :-)   2016-06-21 12:29:50
nova-osapi_compute 0.0.0.0                              internal         enabled    XXX   None
nova-metadata    0.0.0.0                              internal         enabled    XXX   None

More input:

What is OK:

root@angelus ~ # openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                       |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| 2725ef86b8c243cb8800e97efe2e27d2 | RegionOne | nova         | compute      | True    | admin     | http://controller:8774/v2.1/%(tenant_id)s |
| 39cbf1384810485c9360e45e242670db | RegionOne | keystone     | identity     | True    | public    | http://controller:5000/v3                 |
| 528d967ebd324fd38dc4ef61d698c615 | RegionOne | nova         | compute      | True    | internal  | http://controller:8774/v2.1/%(tenant_id)s |
| a55e53e4da504ff68976ca8448b09f47 | RegionOne | glance       | image        | True    | admin     | http://controller:9292                    |
| b5f3b197574842edbce738f967188cdd | RegionOne | glance       | image        | True    | public    | http://controller:9292                    |
| c0314ee49a5b45469fe038854d2ff16c | RegionOne | glance       | image        | True    | internal  | http://controller:9292                    |
| c079956dffb14a8087357bf55692581d | RegionOne | keystone     | identity     | True    | admin     | http://controller:35357/v3                |
| fbbc2cc25a274f2781c5e0b59517bc96 | RegionOne | nova         | compute      | True    | public    | http://controller:8774/v2.1/%(tenant_id)s |
| fdeb937b16294ba5bdfc05b7f98c131e | RegionOne | keystone     | identity     | True    | internal  | http://controller:5000/v3                 |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+

-

root@angelus ~ # openstack catalog list
+----------+----------+--------------------------------------------------------------------------+
| Name     | Type     | Endpoints                                                                |
+----------+----------+--------------------------------------------------------------------------+
| keystone | identity | RegionOne                                                                |
|          |          |   public: http://controller:5000/v3                                      |
|          |          | RegionOne                                                                |
|          |          |   admin: http://controller:35357/v3                                      |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:5000/v3                                    |
|          |          |                                                                          |
| glance   | image    | RegionOne                                                                |
|          |          |   admin: http://controller:9292                                          |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:9292                                         |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:9292                                       |
|          |          |                                                                          |
| nova     | compute  | RegionOne                                                                |
|          |          |   admin: http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d    |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d   |
|          |          |                                                                          |
+----------+----------+--------------------------------------------------------------------------+

-

root@angelus ~ # openstack user list
+----------------------------------+--------+
| ID                               | Name   |
+----------------------------------+--------+
| 13b0ca7e901e447f85e10778d07c1bbe | admin  |
| 870c58ad1ffe4341aa94e84f07bbf055 | demo   |
| db0ef9edc7804d39884dfaa2033526c6 | glance |
| e9c02f8e67c3464abd55773e1ecca96b | nova   |
+----------------------------------+--------+

An image has been successfully uploaded:

root@angelus ~ # openstack image list
+--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| eb81397d-b5e7-4a78-bd8f-764a6bc5d26c | cirros | active |
+--------------------------------------+--------+--------+

Status of 2 services is "XXX":

root@angelus ~ # nova-manage service list
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
DEPRECATED: Use the nova service-* commands from python-novaclient instead or the os-services REST resource. The service subcommand will be removed in the 14.0 release.
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth angelus                              internal         enabled    :-)   2016-06-21 12:29:46
nova-conductor   angelus                              internal         enabled    :-)   2016-06-21 12:29:51
nova-scheduler   angelus                              internal         enabled    :-)   2016-06-21 12:29:50
nova-osapi_compute 0.0.0.0                              internal         enabled    XXX   None
nova-metadata    0.0.0.0                              internal         enabled ...
(more)
edit retag flag offensive close merge delete
add a comment

5 answers

Sort by » oldest newest most voted
1

answered 2016-09-16 08:36:53 -0600

rioyu gravatar image

Hi guys~ I have the same problem before beacuse of the wrong nova password.

Please make sure you have a correct password in /etc/nova/nova.conf

[keystone_authtoken] ... password = NOVA_PASS

Replace NOVA_PASS with the password you chose for the nova user in the Identity service.

openstack user create --domain default --password-prompt nova

Good luck :)

edit flag offensive delete link more
add a comment
0

answered 2016-09-11 09:31:13 -0600

Davidshu gravatar image

Hi, I have same problem

edit flag offensive delete link more

Comments

Do you have a correct nova password in /etc/nova/nova.conf ?

rioyu gravatar imagerioyu ( 2016-09-16 08:36:36 -0600 )edit
add a comment
0

answered 2016-06-21 15:38:39 -0600

Asvin gravatar image

check the nova-api service status. Is it up and running without any errors ?

Check the rabbitmq config on nova.conf file on controllers

edit flag offensive delete link more

Comments

can you plz paste the nova config file on controller and compute node

I have seen this before, I m pretty sure 99% its related to keystone auth.

Asvin gravatar imageAsvin ( 2016-06-22 12:05:53 -0600 )edit

Configuration files added to the answer.

pascalnok gravatar imagepascalnok ( 2016-06-30 10:55:43 -0600 )edit

Is your network up and running? what's the output of neutron agent-list? In my cloud the nova endpoint is http://control1.cloud.hh.nde.ag:8774/v2/%(tenant_id)s and not v2.1 like you have, not sure if that's a possible problem...

eblock gravatar imageeblock ( 2016-07-01 02:22:11 -0600 )edit

In the document (http://docs.openstack.org/mitaka/install-guide-rdo (http://docs.openstack.org/mitaka/inst...)), the verify operation (openstack compute service list) is just before the network chapter.

root@angelus ~ # neutron agent-list
public endpoint for network service not found
pascalnok gravatar imagepascalnok ( 2016-07-06 09:16:52 -0600 )edit

In the same document -> Install and configure controller node -> Create the Compute service API endpoints, version is 2.1.

pascalnok gravatar imagepascalnok ( 2016-07-06 09:17:14 -0600 )edit
add a comment
0

answered 2016-06-22 09:47:16 -0600

pascalnok gravatar image

updated 2016-06-30 10:54:35 -0600

Hi,

Machine rebooted. Status is the same.

The service openstack-nova-api is running:

root@angelus ~ # systemctl status openstack-nova-api
● openstack-nova-api.service - OpenStack Nova API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-22 15:53:06 CEST; 21min ago
 Main PID: 1008 (nova-api)
   CGroup: /system.slice/openstack-nova-api.service
           ├─1008 /usr/bin/python2 /usr/bin/nova-api
           ├─2681 /usr/bin/python2 /usr/bin/nova-api
           ├─2682 /usr/bin/python2 /usr/bin/nova-api
           ├─2683 /usr/bin/python2 /usr/bin/nova-api
           ├─2684 /usr/bin/python2 /usr/bin/nova-api
           ├─2711 /usr/bin/python2 /usr/bin/nova-api
           ├─2712 /usr/bin/python2 /usr/bin/nova-api
           ├─2713 /usr/bin/python2 /usr/bin/nova-api
           └─2714 /usr/bin/python2 /usr/bin/nova-api

Jun 22 15:52:55 angelus systemd[1]: Starting OpenStack Nova API Server...
Jun 22 15:53:05 angelus sudo[2685]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c
Jun 22 15:53:05 angelus sudo[2698]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c
Jun 22 15:53:06 angelus systemd[1]: Started OpenStack Nova API Server.

No error in the log file, last lines are:

root@angelus ~ # tail -14 /var/log/nova/nova-api.log
2016-06-22 15:53:05.507 1008 INFO nova.api.openstack [-] Loaded extensions: ['extensions', 'flavors', 'image-metadata', 'image-size', 'images', 'ips', 'limits', 'os-access-ips', 'os-admin-actions', 'os-admin-password', 'os-agents', 'os-aggregates', 'os-assisted-volume-snapshots', 'os-attach-interfaces', 'os-availability-zone', 'os-baremetal-nodes', 'os-block-device-mapping', 'os-cells', 'os-certificates', 'os-cloudpipe', 'os-config-drive', 'os-console-auth-tokens', 'os-console-output', 'os-consoles', 'os-create-backup', 'os-deferred-delete', 'os-disk-config', 'os-evacuate', 'os-extended-availability-zone', 'os-extended-server-attributes', 'os-extended-status', 'os-extended-volumes', 'os-fixed-ips', 'os-flavor-access', 'os-flavor-extra-specs', 'os-flavor-manage', 'os-flavor-rxtx', 'os-floating-ip-dns', 'os-floating-ip-pools', 'os-floating-ips', 'os-floating-ips-bulk', 'os-fping', 'os-hide-server-addresses', 'os-hosts', 'os-hypervisors', 'os-instance-actions', 'os-instance-usage-audit-log', 'os-keypairs', 'os-lock-server', 'os-migrate-server', 'os-migrations', 'os-multinic', 'os-multiple-create', 'os-networks', 'os-networks-associate', 'os-pause-server', 'os-personality', 'os-preserve-ephemeral-rebuild', 'os-quota-class-sets', 'os-quota-sets', 'os-remote-consoles', 'os-rescue', 'os-scheduler-hints', 'os-security-group-default-rules', 'os-security-groups', 'os-server-diagnostics', 'os-server-external-events', 'os-server-groups', 'os-server-password', 'os-server-usage', 'os-services', 'os-shelve', 'os-simple-tenant-usage', 'os-suspend-server', 'os-tenant-networks', 'os-used-limits', 'os-user-data', 'os-virtual-interfaces', 'os-volumes', 'server-metadata', 'server-migrations', 'servers', 'versions']
2016-06-22 15:53:05.526 1008 INFO nova.wsgi [-] osapi_compute listening on 0.0.0.0:8774
2016-06-22 15:53:05.526 1008 INFO oslo_service.service [-] Starting 4 workers
2016-06-22 15:53:05.546 1008 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2016-06-22 15:53:05.776 2684 INFO nova.osapi_compute.wsgi.server [req-d4874885-6fa5-4375-9fb0-499a22e752e8 - - - - -] (2684) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:05.776 2681 INFO nova.osapi_compute.wsgi.server [req-2d99c158-c067-4450-9d16-bd32dc9900a5 - - - - -] (2681) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:05.786 2682 INFO nova.osapi_compute.wsgi.server [req-5a18f193-90e6-4da5-972b-5f0b6257ce9b - - - - -] (2682) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:05.791 2683 INFO nova.osapi_compute.wsgi.server [req-c6d431bd-f95e-43a1-9ce1-e022d5887f2d - - - - -] (2683) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:06.176 1008 INFO nova.wsgi [-] metadata listening on 0.0.0.0:8775
2016-06-22 15:53:06.176 1008 INFO oslo_service.service [-] Starting 4 workers
2016-06-22 15:53:06.420 2711 INFO nova.metadata.wsgi.server [req-864ce7b3-81d3-4274-a74d-7d0c43c6830e - - - - -] (2711) wsgi starting up on http://0.0.0.0:8775/
2016-06-22 15:53:06.424 2713 INFO ...
(more)
edit flag offensive delete link more
add a comment
0

answered 2018-07-16 03:29:06 -0600

OpenHead gravatar image

I use Pike version. I encountered the same problem. Here is my solution after googling it , and took me 8 hours.

In general, this is caused by a miss typed NOVA password in most cases (searching the internet for the same error)

1. Verify the error in formation in /var/log/nova/nova-api.log

   $ tail -14 /var/log/nova/nova-api.log

You may find the error information as like

 - 2018-07-16 04:16:47.355 3156 WARNING keystonemiddleware.auth_token[req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] Identity response: {"error": {"message": "The request you have made requires   authentication.", "code": 401,   
   "title": "Unauthorized"}}:   *Unauthorized: The request you have made requires authentication. (HTTP  
   401) (Request-ID:   req-1e2d504f-fda2-48c4-abb9-9d81953d15aa)*

 - -2018-07-16 04:16:55.998 3156 WARNING    keystonemiddleware.auth_token   
   [req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] Identity response: {"error": {"message": "The request   
   you have made requires   
   authentication.", "code": 401,   
   "title": "Unauthorized"}}:   
   Unauthorized: The request you have   
   made requires authentication. (HTTP  
   401) (Request-ID:   
   req-9d625573-b5d9-40b1-8a19-267653246958)



 - -2018-07-16 04:16:55.998 3156 CRITICAL   
   keystonemiddleware.auth_token   
   [req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] Unable to validate token: Identity server rejected   
   authorization necessary to fetch   
   token data: ServiceError: Identity   
   server rejected authorization   
   necessary to fetch token data
 - 2018-07-16 04:16:56.001 3156 INFO
   nova.osapi_compute.wsgi.server
   [req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] 172.168.1.11 "GET /v2.1/os-services HTTP/1.1" status:
   503 len: 450 time: 17.0881312

"ServiceError: Identity server rejected authorization necessary to fetch token data" means Keystone authentication failed.

2 Verify your password in /etc/nova/nova.conf

[keystone_authentication]

    password = NOVA_PASS  
    #you use your one to replace NOVA_PASS

The password should be as the same as you input in below command 

    $ openstack user create --domain default --password-prompt nova
    # refer to https://docs.openstack.org/nova/pike/install/controller-install-obs.html

if you cannot double confirm this password, then reset it. to do this

3. Reset nova user password

> $openstack user set --password NOVA_PASS nova

Add the admin role to the nova user again

 $openstack role add --project service --user nova admin


> # https://ask.openstack.org/en/question/95124/mitaka-glance-503-service-unavailable-the-server-is-currently-unavailable-please-try-again-at-a-later-time-http-503/

Hope it can help! it fixed my problem.

edit flag offensive delete link more
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2016-06-21 08:18:11 -0600

Seen: 14,641 times

Last updated: Sep 16 '16

Related questions

Overcommitting of CPUs on Qemu?

neutron client authentication failed

Add -cpu kvm=off flag

Instance stuck booting from Hard Disk

ping running instance

Failure prepping block device

slow snapshot [closed]

Disk I/O scheduling

Injecting password to VM

Launching instances fails with "Error from last host message"