Nova List shows instances but dashboard does not show any instance

asked 2017-08-03 21:40:59 -0500

gaurav.mittal gravatar image

updated 2017-08-03 22:18:07 -0500

Hi,

I am using Openstack OCATA and i am seeing a veired issue. IN the dashboard page in instances tab i do not see any instances. Sometime on refreshing the page it lists the instances and immediately it vanishes from the page.

But when I run the CLI command Nova list I see the instances.

[root@openstack ~(keystone_admin)]# nova list
+--------------------------------------+----------------+--------+------------+-------------+----------+
| ID                                   | Name           | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+----------+
| 660b33fd-6f3f-4505-bba6-eccd35fe3e6e | vm1-dns-server | BUILD  | scheduling | NOSTATE     |          |
| 93cb7637-10f3-4dd9-b1b2-66f236fee6c6 | vm1-vid        | BUILD  | scheduling | NOSTATE     |          |
+--------------------------------------+----------------+--------+------------+-------------+----------+
[root@openstack ~(keystone_admin)]#

I also see that the all the services are running.

[root@openstack ~(keystone_admin)]# nova service-list
+----+------------------+-----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host      | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+-----------+----------+---------+-------+----------------------------+-----------------+
| 6  | nova-cert        | openstack | internal | enabled | up    | 2017-08-04T02:23:49.000000 | -               |
| 7  | nova-conductor   | openstack | internal | enabled | up    | 2017-08-04T02:23:46.000000 | -               |
| 9  | nova-scheduler   | openstack | internal | enabled | up    | 2017-08-04T02:23:48.000000 | -               |
| 10 | nova-consoleauth | openstack | internal | enabled | up    | 2017-08-04T02:23:47.000000 | -               |
| 11 | nova-compute     | openstack | nova     | enabled | up    | 2017-08-04T02:23:52.000000 | -               |
+----+------------------+-----------+----------+---------+-------+----------------------------+-----------------+
[root@openstack ~(keystone_admin)]#

I tried to make the state as active that also fails.

[root@openstack ~(keystone_admin)]# nova reset-state --active 660b33fd-6f3f-4505-bba6-eccd35fe3e6e
Reset state for server 660b33fd-6f3f-4505-bba6-eccd35fe3e6e failed: No server with a name or ID of '660b33fd-6f3f-4505-bba6-eccd35fe3e6e' exists.
ERROR (CommandError): Unable to reset the state for the specified server(s).
[root@openstack ~(keystone_admin)]#

I try to delete the instance that also fails.

[root@openstack ~(keystone_admin)]# nova delete 660b33fd-6f3f-4505-bba6-eccd35fe3e6e
No server with a name or ID of '660b33fd-6f3f-4505-bba6-eccd35fe3e6e' exists.
ERROR (CommandError): Unable to delete the specified server(s).
[root@openstack ~(keystone_admin)]#

Its a fresh installation of open stack where there are no changes done to the open stack.

Below is what I see in nova-api.log file.

2017-08-04 08:10:49.463 7172 INFO nova.osapi_compute.wsgi.server [req-ef41af4b-750a-420e-a3dd-039b1230ef66 5e4ba6d6245a443984e2a77c3dce44e4 fd0d6b9655a74a439e77b40a4705b6b9 - default default] 10.211.2.21 "GET /v2.1/fd0d6b9655a74a439e77b40a4705b6b9 HTTP/1.1" status: 404 len: 331 time: 0.2152889
2017-08-04 08:10:49.480 7172 INFO nova.osapi_compute.wsgi.server [req-5f83745f-5f57-4edc-ae27-4454d75bb137 5e4ba6d6245a443984e2a77c3dce44e4 fd0d6b9655a74a439e77b40a4705b6b9 - default default] 10.211.2.21 "GET /v2.1/ HTTP/1.1" status: 200 len: 718 time: 0.0143678
2017-08-04 08:10:49.784 7172 INFO nova.api.openstack.wsgi [req-45177972-55b1-43eb-ad00-8777e2db43ad 5e4ba6d6245a443984e2a77c3dce44e4 fd0d6b9655a74a439e77b40a4705b6b9 - default default] HTTP exception thrown: Instance 660b33fd-6f3f-4505-bba6-eccd35fe3e6e could not be found.
2017-08-04 08:10:49.786 7172 INFO nova.osapi_compute.wsgi.server [req-45177972-55b1-43eb-ad00-8777e2db43ad 5e4ba6d6245a443984e2a77c3dce44e4 fd0d6b9655a74a439e77b40a4705b6b9 - default default] 10.211.2.21 "GET /v2.1/fd0d6b9655a74a439e77b40a4705b6b9/servers/660b33fd-6f3f-4505-bba6-eccd35fe3e6e HTTP/1.1" status: 404 len: 468 time: 0.0527210
2017-08-04 08:10:50.005 7172 INFO nova.osapi_compute.wsgi.server [req-7df76658-f809-48b7-91a6-89087284bef9 5e4ba6d6245a443984e2a77c3dce44e4 fd0d6b9655a74a439e77b40a4705b6b9 - default default] 10.211.2.21 "GET /v2.1/fd0d6b9655a74a439e77b40a4705b6b9/servers?name=660b33fd-6f3f-4505-bba6-eccd35fe3e6e HTTP/1.1" status: 200 len: 349 time: 0.2119131
2017-08-04 08:10:50.208 7172 INFO nova.osapi_compute.wsgi.server [req-425d59a6-cfee-45ba-b622-d3acbc9778a1 5e4ba6d6245a443984e2a77c3dce44e4 fd0d6b9655a74a439e77b40a4705b6b9 - default default] 10.211.2.21 "GET /v2.1/fd0d6b9655a74a439e77b40a4705b6b9/servers?name=660b33fd-6f3f-4505-bba6-eccd35fe3e6e HTTP/1.1" status: 200 len: 349 time: 0.1964431

Any help will be highly appreciated.

Gaurav

edit retag flag offensive close merge delete

Comments

Hi, It seems VMs are stuck during BUILD. Try to stop nova-compute service and then start. Also restart rabbitmq server. What does neutron server log says? please share the output of nova show <InstanceID>

CloudEnthusiast gravatar imageCloudEnthusiast ( 2017-08-03 22:52:27 -0500 )edit

Thanks a lot for the answer. Below is output as requested.

[root@openstack ~(keystone_admin)]# nova show 660b33fd-6f3f-4505-bba6-eccd35fe3e6e
ERROR (CommandError): No server with a name or ID of '660b33fd-6f3f-4505-bba6-eccd35fe3e6e' exists.
[root@openstack ~(keystone_admin)]#
gaurav.mittal gravatar imagegaurav.mittal ( 2017-08-03 23:49:44 -0500 )edit

[root@openstack ~(keystone_admin)]# systemctl restart openstack-nova-compute [root@openstack ~(keystone_admin)]# /sbin/service rabbitmq-server stop Redirecting to /bin/systemctl stop rabbitmq-server.service [root@openstack ~(keystone_admin)]#

gaurav.mittal gravatar imagegaurav.mittal ( 2017-08-03 23:54:15 -0500 )edit

[root@openstack ~(keystone_admin)]# /sbin/service rabbitmq-server start Redirecting to /bin/systemctl start rabbitmq-server.service [root@openstack ~(keystone_admin)]#

gaurav.mittal gravatar imagegaurav.mittal ( 2017-08-03 23:54:32 -0500 )edit

After doing this also i am not able to fix the issue. [root@openstack ~(keystone_admin)]# nova reset-state --active 660b33fd-6f3f-4505-bba6-eccd35fe3e6e Reset state for server 660b33fd-6f3f-4505-bba6-eccd35fe3e6e failed: No server with a name or ID of '660b33fd-6f3f-4505-bba6-eccd35fe3e6e' exists.

gaurav.mittal gravatar imagegaurav.mittal ( 2017-08-03 23:55:17 -0500 )edit

after this nova list still shows the instances in BUILD state ? Looks like a problem with scheduler. Check neutron server logs and nova scheduler logs for any clue.

CloudEnthusiast gravatar imageCloudEnthusiast ( 2017-08-04 00:51:38 -0500 )edit

Have you set debug = True in nova.conf ?? For time being reset the instance status to shut off forcefully. Later run nova delete <instanceID> Then try again to launch instances.

CloudEnthusiast gravatar imageCloudEnthusiast ( 2017-08-04 00:54:45 -0500 )edit

Is it possible that you are in the wrong project when navigating through dashboard? The messages you describe sound like you started the instances in project A, but your dashboard view shows project B. Just a thought to rule that out...

eblock gravatar imageeblock ( 2017-08-04 02:48:22 -0500 )edit

I do not have multiple projects. every thing is running in admin Project. I have not even installed the demo project.

gaurav.mittal gravatar imagegaurav.mittal ( 2017-08-04 03:49:07 -0500 )edit

Is your database connection working properly? Do you see the instances in the db?

eblock gravatar imageeblock ( 2017-08-04 05:15:04 -0500 )edit

Can you please kindly share the commands to check the instances in DB.

gaurav.mittal gravatar imagegaurav.mittal ( 2017-08-04 05:36:24 -0500 )edit