Ask Your Question
0

openstack compute service list: Unknown Error (HTTP 503) / Service Unavailable

asked 2016-06-21 08:18:11 -0500

pascalnok gravatar image

Hi all,

I try to install a controller (following http://docs.openstack.org/mitaka/install-guide-rdo). All is OK until the verification of services components:

root@angelus ~ # openstack compute service list
Unknown Error (HTTP 503) (Request-ID: req-64e7d1fb-63b6-42d5-87f3-414580caf15d)

With --debug option, the error "503 Service Unavailable" is additionally displayed:

REQ: curl -g -i -X GET http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d/os-services ....
Starting new HTTP connection (1): controller
"GET /v2.1/4d5fc54a120c42b9b76ff438c944db0d/os-services HTTP/1.1" 503 100
RESP: [503] Date: Tue, 21 Jun 2016 12:05:37 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 100 X-Compute-Request-Id: req-a6a5e831-c3c4-47f7-ba49-484a506c5340
RESP BODY: 503 Service Unavailable
The server is currently unavailable. Please try again at a later time.

And I can see the status of 2 services (nova-osapi_compute and nova-metadata) returned by nova-manage service list is "XXX":

Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth angelus                              internal         enabled    :-)   2016-06-21 12:29:46
nova-conductor   angelus                              internal         enabled    :-)   2016-06-21 12:29:51
nova-scheduler   angelus                              internal         enabled    :-)   2016-06-21 12:29:50
nova-osapi_compute 0.0.0.0                              internal         enabled    XXX   None
nova-metadata    0.0.0.0                              internal         enabled    XXX   None

More input:

What is OK:

root@angelus ~ # openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                       |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| 2725ef86b8c243cb8800e97efe2e27d2 | RegionOne | nova         | compute      | True    | admin     | http://controller:8774/v2.1/%(tenant_id)s |
| 39cbf1384810485c9360e45e242670db | RegionOne | keystone     | identity     | True    | public    | http://controller:5000/v3                 |
| 528d967ebd324fd38dc4ef61d698c615 | RegionOne | nova         | compute      | True    | internal  | http://controller:8774/v2.1/%(tenant_id)s |
| a55e53e4da504ff68976ca8448b09f47 | RegionOne | glance       | image        | True    | admin     | http://controller:9292                    |
| b5f3b197574842edbce738f967188cdd | RegionOne | glance       | image        | True    | public    | http://controller:9292                    |
| c0314ee49a5b45469fe038854d2ff16c | RegionOne | glance       | image        | True    | internal  | http://controller:9292                    |
| c079956dffb14a8087357bf55692581d | RegionOne | keystone     | identity     | True    | admin     | http://controller:35357/v3                |
| fbbc2cc25a274f2781c5e0b59517bc96 | RegionOne | nova         | compute      | True    | public    | http://controller:8774/v2.1/%(tenant_id)s |
| fdeb937b16294ba5bdfc05b7f98c131e | RegionOne | keystone     | identity     | True    | internal  | http://controller:5000/v3                 |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+

-

root@angelus ~ # openstack catalog list
+----------+----------+--------------------------------------------------------------------------+
| Name     | Type     | Endpoints                                                                |
+----------+----------+--------------------------------------------------------------------------+
| keystone | identity | RegionOne                                                                |
|          |          |   public: http://controller:5000/v3                                      |
|          |          | RegionOne                                                                |
|          |          |   admin: http://controller:35357/v3                                      |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:5000/v3                                    |
|          |          |                                                                          |
| glance   | image    | RegionOne                                                                |
|          |          |   admin: http://controller:9292                                          |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:9292                                         |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:9292                                       |
|          |          |                                                                          |
| nova     | compute  | RegionOne                                                                |
|          |          |   admin: http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d    |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:8774/v2.1/4d5fc54a120c42b9b76ff438c944db0d   |
|          |          |                                                                          |
+----------+----------+--------------------------------------------------------------------------+

-

root@angelus ~ # openstack user list
+----------------------------------+--------+
| ID                               | Name   |
+----------------------------------+--------+
| 13b0ca7e901e447f85e10778d07c1bbe | admin  |
| 870c58ad1ffe4341aa94e84f07bbf055 | demo   |
| db0ef9edc7804d39884dfaa2033526c6 | glance |
| e9c02f8e67c3464abd55773e1ecca96b | nova   |
+----------------------------------+--------+

An image has been successfully uploaded:

root@angelus ~ # openstack image list
+--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| eb81397d-b5e7-4a78-bd8f-764a6bc5d26c | cirros | active |
+--------------------------------------+--------+--------+

Status of 2 services is "XXX":

root@angelus ~ # nova-manage service list
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
DEPRECATED: Use the nova service-* commands from python-novaclient instead or the os-services REST resource. The service subcommand will be removed in the 14.0 release.
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth angelus                              internal         enabled    :-)   2016-06-21 12:29:46
nova-conductor   angelus                              internal         enabled    :-)   2016-06-21 12:29:51
nova-scheduler   angelus                              internal         enabled    :-)   2016-06-21 12:29:50
nova-osapi_compute 0.0.0.0                              internal         enabled    XXX   None
nova-metadata    0.0.0.0                              internal         enabled ...
(more)
edit retag flag offensive close merge delete
add a comment

5 answers

Sort by » oldest newest most voted
1

answered 2016-09-16 08:36:53 -0500

rioyu gravatar image

Hi guys~ I have the same problem before beacuse of the wrong nova password.

Please make sure you have a correct password in /etc/nova/nova.conf

[keystone_authtoken] ... password = NOVA_PASS

Replace NOVA_PASS with the password you chose for the nova user in the Identity service.

openstack user create --domain default --password-prompt nova

Good luck :)

edit flag offensive delete link more
add a comment
0

answered 2016-09-11 09:31:13 -0500

Davidshu gravatar image

Hi, I have same problem

edit flag offensive delete link more

Comments

Do you have a correct nova password in /etc/nova/nova.conf ?

rioyu gravatar imagerioyu ( 2016-09-16 08:36:36 -0500 )edit
add a comment
0

answered 2016-06-21 15:38:39 -0500

Asvin gravatar image

check the nova-api service status. Is it up and running without any errors ?

Check the rabbitmq config on nova.conf file on controllers

edit flag offensive delete link more

Comments

can you plz paste the nova config file on controller and compute node

I have seen this before, I m pretty sure 99% its related to keystone auth.

Asvin gravatar imageAsvin ( 2016-06-22 12:05:53 -0500 )edit

Configuration files added to the answer.

pascalnok gravatar imagepascalnok ( 2016-06-30 10:55:43 -0500 )edit

Is your network up and running? what's the output of neutron agent-list? In my cloud the nova endpoint is http://control1.cloud.hh.nde.ag:8774/v2/%(tenant_id)s and not v2.1 like you have, not sure if that's a possible problem...

eblock gravatar imageeblock ( 2016-07-01 02:22:11 -0500 )edit

In the document (http://docs.openstack.org/mitaka/install-guide-rdo (http://docs.openstack.org/mitaka/inst...)), the verify operation (openstack compute service list) is just before the network chapter.

root@angelus ~ # neutron agent-list
public endpoint for network service not found
pascalnok gravatar imagepascalnok ( 2016-07-06 09:16:52 -0500 )edit

In the same document -> Install and configure controller node -> Create the Compute service API endpoints, version is 2.1.

pascalnok gravatar imagepascalnok ( 2016-07-06 09:17:14 -0500 )edit
add a comment
0

answered 2016-06-22 09:47:16 -0500

pascalnok gravatar image

updated 2016-06-30 10:54:35 -0500

Hi,

Machine rebooted. Status is the same.

The service openstack-nova-api is running:

root@angelus ~ # systemctl status openstack-nova-api
● openstack-nova-api.service - OpenStack Nova API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-22 15:53:06 CEST; 21min ago
 Main PID: 1008 (nova-api)
   CGroup: /system.slice/openstack-nova-api.service
           ├─1008 /usr/bin/python2 /usr/bin/nova-api
           ├─2681 /usr/bin/python2 /usr/bin/nova-api
           ├─2682 /usr/bin/python2 /usr/bin/nova-api
           ├─2683 /usr/bin/python2 /usr/bin/nova-api
           ├─2684 /usr/bin/python2 /usr/bin/nova-api
           ├─2711 /usr/bin/python2 /usr/bin/nova-api
           ├─2712 /usr/bin/python2 /usr/bin/nova-api
           ├─2713 /usr/bin/python2 /usr/bin/nova-api
           └─2714 /usr/bin/python2 /usr/bin/nova-api

Jun 22 15:52:55 angelus systemd[1]: Starting OpenStack Nova API Server...
Jun 22 15:53:05 angelus sudo[2685]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c
Jun 22 15:53:05 angelus sudo[2698]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c
Jun 22 15:53:06 angelus systemd[1]: Started OpenStack Nova API Server.

No error in the log file, last lines are:

root@angelus ~ # tail -14 /var/log/nova/nova-api.log
2016-06-22 15:53:05.507 1008 INFO nova.api.openstack [-] Loaded extensions: ['extensions', 'flavors', 'image-metadata', 'image-size', 'images', 'ips', 'limits', 'os-access-ips', 'os-admin-actions', 'os-admin-password', 'os-agents', 'os-aggregates', 'os-assisted-volume-snapshots', 'os-attach-interfaces', 'os-availability-zone', 'os-baremetal-nodes', 'os-block-device-mapping', 'os-cells', 'os-certificates', 'os-cloudpipe', 'os-config-drive', 'os-console-auth-tokens', 'os-console-output', 'os-consoles', 'os-create-backup', 'os-deferred-delete', 'os-disk-config', 'os-evacuate', 'os-extended-availability-zone', 'os-extended-server-attributes', 'os-extended-status', 'os-extended-volumes', 'os-fixed-ips', 'os-flavor-access', 'os-flavor-extra-specs', 'os-flavor-manage', 'os-flavor-rxtx', 'os-floating-ip-dns', 'os-floating-ip-pools', 'os-floating-ips', 'os-floating-ips-bulk', 'os-fping', 'os-hide-server-addresses', 'os-hosts', 'os-hypervisors', 'os-instance-actions', 'os-instance-usage-audit-log', 'os-keypairs', 'os-lock-server', 'os-migrate-server', 'os-migrations', 'os-multinic', 'os-multiple-create', 'os-networks', 'os-networks-associate', 'os-pause-server', 'os-personality', 'os-preserve-ephemeral-rebuild', 'os-quota-class-sets', 'os-quota-sets', 'os-remote-consoles', 'os-rescue', 'os-scheduler-hints', 'os-security-group-default-rules', 'os-security-groups', 'os-server-diagnostics', 'os-server-external-events', 'os-server-groups', 'os-server-password', 'os-server-usage', 'os-services', 'os-shelve', 'os-simple-tenant-usage', 'os-suspend-server', 'os-tenant-networks', 'os-used-limits', 'os-user-data', 'os-virtual-interfaces', 'os-volumes', 'server-metadata', 'server-migrations', 'servers', 'versions']
2016-06-22 15:53:05.526 1008 INFO nova.wsgi [-] osapi_compute listening on 0.0.0.0:8774
2016-06-22 15:53:05.526 1008 INFO oslo_service.service [-] Starting 4 workers
2016-06-22 15:53:05.546 1008 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2016-06-22 15:53:05.776 2684 INFO nova.osapi_compute.wsgi.server [req-d4874885-6fa5-4375-9fb0-499a22e752e8 - - - - -] (2684) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:05.776 2681 INFO nova.osapi_compute.wsgi.server [req-2d99c158-c067-4450-9d16-bd32dc9900a5 - - - - -] (2681) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:05.786 2682 INFO nova.osapi_compute.wsgi.server [req-5a18f193-90e6-4da5-972b-5f0b6257ce9b - - - - -] (2682) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:05.791 2683 INFO nova.osapi_compute.wsgi.server [req-c6d431bd-f95e-43a1-9ce1-e022d5887f2d - - - - -] (2683) wsgi starting up on http://0.0.0.0:8774/
2016-06-22 15:53:06.176 1008 INFO nova.wsgi [-] metadata listening on 0.0.0.0:8775
2016-06-22 15:53:06.176 1008 INFO oslo_service.service [-] Starting 4 workers
2016-06-22 15:53:06.420 2711 INFO nova.metadata.wsgi.server [req-864ce7b3-81d3-4274-a74d-7d0c43c6830e - - - - -] (2711) wsgi starting up on http://0.0.0.0:8775/
2016-06-22 15:53:06.424 2713 INFO ...
(more)
edit flag offensive delete link more
add a comment
0

answered 2018-07-16 03:29:06 -0500

OpenHead gravatar image

I use Pike version. I encountered the same problem. Here is my solution after googling it , and took me 8 hours.

In general, this is caused by a miss typed NOVA password in most cases (searching the internet for the same error)

1. Verify the error in formation in /var/log/nova/nova-api.log

   $ tail -14 /var/log/nova/nova-api.log

You may find the error information as like

 - 2018-07-16 04:16:47.355 3156 WARNING keystonemiddleware.auth_token[req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] Identity response: {"error": {"message": "The request you have made requires   authentication.", "code": 401,   
   "title": "Unauthorized"}}:   *Unauthorized: The request you have made requires authentication. (HTTP  
   401) (Request-ID:   req-1e2d504f-fda2-48c4-abb9-9d81953d15aa)*

 - -2018-07-16 04:16:55.998 3156 WARNING    keystonemiddleware.auth_token   
   [req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] Identity response: {"error": {"message": "The request   
   you have made requires   
   authentication.", "code": 401,   
   "title": "Unauthorized"}}:   
   Unauthorized: The request you have   
   made requires authentication. (HTTP  
   401) (Request-ID:   
   req-9d625573-b5d9-40b1-8a19-267653246958)



 - -2018-07-16 04:16:55.998 3156 CRITICAL   
   keystonemiddleware.auth_token   
   [req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] Unable to validate token: Identity server rejected   
   authorization necessary to fetch   
   token data: ServiceError: Identity   
   server rejected authorization   
   necessary to fetch token data
 - 2018-07-16 04:16:56.001 3156 INFO
   nova.osapi_compute.wsgi.server
   [req-65205ebf-763f-4733-860b-a1d65d68e35f - - - - -] 172.168.1.11 "GET /v2.1/os-services HTTP/1.1" status:
   503 len: 450 time: 17.0881312

"ServiceError: Identity server rejected authorization necessary to fetch token data" means Keystone authentication failed.

2 Verify your password in /etc/nova/nova.conf

[keystone_authentication]

    password = NOVA_PASS  
    #you use your one to replace NOVA_PASS

The password should be as the same as you input in below command 

    $ openstack user create --domain default --password-prompt nova
    # refer to https://docs.openstack.org/nova/pike/install/controller-install-obs.html

if you cannot double confirm this password, then reset it. to do this

3. Reset nova user password

> $openstack user set --password NOVA_PASS nova

Add the admin role to the nova user again

 $openstack role add --project service --user nova admin


> # https://ask.openstack.org/en/question/95124/mitaka-glance-503-service-unavailable-the-server-is-currently-unavailable-please-try-again-at-a-later-time-http-503/

Hope it can help! it fixed my problem.

edit flag offensive delete link more
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2016-06-21 08:18:11 -0500

Seen: 13,332 times

Last updated: Sep 16 '16

Related questions

How to delete network

Change default (path) emulator for libvirt

Disk Quota for Projects

Standalone mode (no network)

Hypervisor summary shows incorrect total storage

network bandwidth based on flavor extra specs are not working

Modifying base .xml templates for Windows VM

Snapshot of Instances is Queued forever

nova pause return 'n/a (HTTP 404)'

How to get quota usage from nova API?