Revision history [back]

click to hide/show revision 1
initial version

Error: Unable to retrieve instances

Hi,

We're using OpenStack with just 3 nodes at the moment (1 Horizon, 1 Neutron and 1 Nova). Installation was done via Packstack.

Things were working well (although some of the errors in Horizon were already there) until a power loss left us in this state:

In Horizon, when trying to list the Instances, we get: Error: Unable to retrieve instances. Something "similar" happens, for example. Under Volumes - "Error: Unable to retrieve volume/instance attachment information" Under Overview - "Error: Unable to retrieve attachment information." and "Error: Unable to retrieve network quota information."

This happens in several projects (all where there are some instances).

After trying to look in logs and the several servers, I've noticed that neutron-server service is not running and doesn't start:

[root@neutron-00 ~]# systemctl start neutron-server.service Job for neutron-server.service failed because the control process exited with error code. See "systemctl status neutron-server.service" and "journalctl -xe" for details.

[root@neutron-00 ~]# systemctl status neutron-server.service ● neutron-server.service - OpenStack Neutron Server Loaded: loaded (/usr/lib/systemd/system/neutron-server.service; disabled; vendor preset: disabled) Active: activating (start) since Wed 2020-06-24 16:11:33 BST; 3ms ago Main PID: 28634 (neutron-server) CGroup: /system.slice/neutron-server.service └─28634 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutr...

I've tried starting some instances via CLI, but can't even see the instances (I know they are there, since I've queried the nova MySQL database and I can see them there):

# nova list +----+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +----+------+--------+------------+-------------+----------+ +----+------+--------+------------+-------------+----------+

# neutron net-list Unable to establish connection to http://xx.xx.12.10:9696/v2.0/networks.json: HTTPConnectionPool(host='xx.xx.12.10', port=9696): Max retries exceeded with url: /v2.0/networks.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.httpconnection object="" at="" 0x2a1cdd0="">: Failed to establish a new connection: [Errno 111] Connection refused',))

In Neutron:

# tail /var/log/neutron/server.log 2020-06-24 16:15:58.667 31726 ERROR neutron File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception 2020-06-24 16:15:58.667 31726 ERROR neutron util.raise_from_cause(newraise, exc_info) 2020-06-24 16:15:58.667 31726 ERROR neutron File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2020-06-24 16:15:58.667 31726 ERROR neutron reraise(type(exception), exception, tb=exc_tb) 2020-06-24 16:15:58.667 31726 ERROR neutron File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2020-06-24 16:15:58.667 31726 ERROR neutron context) 2020-06-24 16:15:58.667 31726 ERROR neutron File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2020-06-24 16:15:58.667 31726 ERROR neutron cursor.execute(statement, parameters) 2020-06-24 16:15:58.667 31726 ERROR neutron DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations'] 2020-06-24 16:15:58.667 31726 ERROR neutron

Would anyone be kind enough to suggest something we can try to get Neutron, then the rest, going?

Thank you!

click to hide/show revision 2
None

Error: Unable to retrieve instances

Hi,

We're using OpenStack with just 3 nodes at the moment (1 Horizon, 1 Neutron and 1 Nova). Installation was done via Packstack.

Things were working well (although some of the errors in Horizon were already there) until a power loss left us in this state:

In Horizon, when trying to list the Instances, we get:

Error: Unable to retrieve instances. instances.

Something "similar" happens, for example. Under Volumes - Volumes:

"Error: Unable to retrieve volume/instance attachment information" information"

Under Overview - Overview:

"Error: Unable to retrieve attachment information." and "Error: Unable to retrieve network quota information."

This happens in several projects (all where there are some instances).

After trying to look in logs and the several servers, I've noticed that neutron-server service is not running and doesn't start:

[root@neutron-00 ~]# systemctl start neutron-server.service
Job for neutron-server.service failed because the control process exited with error code. See "systemctl status neutron-server.service" and "journalctl -xe" for details.

details. [root@neutron-00 ~]# systemctl status neutron-server.service ● neutron-server.service - OpenStack Neutron Server Loaded: loaded (/usr/lib/systemd/system/neutron-server.service; disabled; vendor preset: disabled) Active: activating (start) since Wed 2020-06-24 16:11:33 BST; 3ms ago Main PID: 28634 (neutron-server) CGroup: /system.slice/neutron-server.service └─28634 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutr...

/etc/neutron/neutr...

I've tried starting some instances via CLI, but can't even see the instances (I know they are there, since I've queried the nova MySQL database and I can see them there):

# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

+----+------+--------+------------+-------------+----------+ # neutron net-list Unable to establish connection to http://xx.xx.12.10:9696/v2.0/networks.json: HTTPConnectionPool(host='xx.xx.12.10', port=9696): Max retries exceeded with url: /v2.0/networks.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.httpconnection object="" at="" 0x2a1cdd0="">: NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2a1cdd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

refused',))

In Neutron:

 # tail /var/log/neutron/server.log 
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2020-06-24 16:15:58.667 31726 ERROR neutron     util.raise_from_cause(newraise, exc_info)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2020-06-24 16:15:58.667 31726 ERROR neutron     reraise(type(exception), exception, tb=exc_tb)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2020-06-24 16:15:58.667 31726 ERROR neutron     context)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2020-06-24 16:15:58.667 31726 ERROR neutron     cursor.execute(statement, parameters)
2020-06-24 16:15:58.667 31726 ERROR neutron DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations']
2020-06-24 16:15:58.667 31726 ERROR neutron 

neutron

Would anyone be kind enough to suggest something we can try to get Neutron, then the rest, going?

Thank you!