Error: Unable to retrieve instances

asked 2020-06-24 10:34:19 -0500

jlmorgado gravatar image

updated 2020-06-25 06:20:43 -0500

eblock gravatar image

Hi,

We're using OpenStack with just 3 nodes at the moment (1 Horizon, 1 Neutron and 1 Nova). Installation was done via Packstack.

Things were working well (although some of the errors in Horizon were already there) until a power loss left us in this state:

In Horizon, when trying to list the Instances, we get:

Error: Unable to retrieve instances.

Something "similar" happens, for example. Under Volumes:

"Error: Unable to retrieve volume/instance attachment information"

Under Overview:

"Error: Unable to retrieve attachment information." and "Error: Unable to retrieve network quota information."

This happens in several projects (all where there are some instances).

After trying to look in logs and the several servers, I've noticed that neutron-server service is not running and doesn't start:

[root@neutron-00 ~]# systemctl start neutron-server.service
Job for neutron-server.service failed because the control process exited with error code. See "systemctl status neutron-server.service" and "journalctl -xe" for details.

[root@neutron-00 ~]# systemctl status neutron-server.service
● neutron-server.service - OpenStack Neutron Server
   Loaded: loaded (/usr/lib/systemd/system/neutron-server.service; disabled; vendor preset: disabled)
   Active: activating (start) since Wed 2020-06-24 16:11:33 BST; 3ms ago
 Main PID: 28634 (neutron-server)
   CGroup: /system.slice/neutron-server.service
           └─28634 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutr...

I've tried starting some instances via CLI, but can't even see the instances (I know they are there, since I've queried the nova MySQL database and I can see them there):

# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

 # neutron net-list
Unable to establish connection to http://xx.xx.12.10:9696/v2.0/networks.json: HTTPConnectionPool(host='xx.xx.12.10', port=9696): Max retries exceeded with url: /v2.0/networks.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2a1cdd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

In Neutron:

 # tail /var/log/neutron/server.log 
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2020-06-24 16:15:58.667 31726 ERROR neutron     util.raise_from_cause(newraise, exc_info)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2020-06-24 16:15:58.667 31726 ERROR neutron     reraise(type(exception), exception, tb=exc_tb)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2020-06-24 16:15:58.667 31726 ERROR neutron     context)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2020-06-24 16:15:58.667 31726 ERROR neutron     cursor.execute(statement, parameters)
2020-06-24 16:15:58.667 31726 ERROR neutron DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations']
2020-06-24 16:15:58.667 31726 ERROR neutron ...
(more)
edit retag flag offensive close merge delete

Comments

You can’t see instances because nova doesn’t start, same goes for the other services because they all rely on the database. You write you could query the db but the errors indicate that it’s not running or in an inconsistent state after the outage.

eblock gravatar imageeblock ( 2020-06-25 00:39:23 -0500 )edit

1 answer

Sort by » oldest newest most voted
0

answered 2020-06-25 07:07:08 -0500

jlmorgado gravatar image

Hi eblock, thank you for your reply.

When I run:

# openstack-status
== Nova services ==
openstack-nova-api:                     active
openstack-nova-compute:                 active
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-cert:                    active
openstack-nova-conductor:               active
openstack-nova-console:                 inactive  (disabled on boot)
openstack-nova-consoleauth:             active
openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active
== Keystone service ==
openstack-keystone:                     inactive  (disabled on boot)
== Horizon service ==
openstack-dashboard:                    active
== neutron services ==
neutron-server:                         activating
neutron-dhcp-agent:                     inactive  (disabled on boot)
neutron-l3-agent:                       inactive  (disabled on boot)
neutron-metadata-agent:                 inactive  (disabled on boot)
neutron-openvswitch-agent:              active

I can see nova services seem to be running... what's still activating (and won't change) is the neutron-server.

Thank you, J.

edit flag offensive delete link more

Comments

As already mentioned, your neutron database seems to be corrupted. Either fix it manually or import a backup, otherwise neutron won't start. Please only manipulate the database if you really know what you're doing! And before doing so, save another backup of the current state to be able to roll back

eblock gravatar imageeblock ( 2020-06-29 02:03:54 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2020-06-24 10:34:19 -0500

Seen: 87 times

Last updated: Jun 25