Unable to complete tasks with instances

asked 2020-05-29 09:20:36 -0500

dgolive gravatar image

updated 2020-05-30 09:38:35 -0500

Hello everyone, I'm getting this error message when try to use "openstack server list" or "openstack server create" command and as well dashboard not open.

$ openstack server list Unable to establish connection to http://172.18.181.51:8774/v2.1/servers/detail (http://172.18.181.51:8774/v2.1/server...): ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

Sometimes it's possible, but very slow almost 1m4.283s.

$ openstack server create --flavor l1.medium1 --image CENTOS-7.6-x86_64-BASE --key-name key1 --nic net-id=demo-net-1 --boot-from-volume 15 demo-centos-dan-3 Unable to establish connection to http://172.18.181.51:8774/v2.1/servers: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

rabbitmq log: =ERROR REPORT==== 29-May-2020::13:54:13 === closing AMQP connection <0.15334.55> (172.18.181.10:46822 -> 172.18.181.37:5672 - mod_wsgi:20920:60886abb-a7c2-43d5-a3b1-3e3de001bf9e): missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 29-May-2020::13:54:14 === closing AMQP connection <0.15341.55> (172.18.181.49:38114 -> 172.18.181.37:5672 - mod_wsgi:13259:4ca0a642-e344-4afe-88d3-b81a1f0b12eb): missed heartbeats from client, timeout: 60s

controller, nova log: 2020-05-29 14:05:47.810 13263 ERROR oslo.messaging._drivers.impl_rabbit [-] [59a3e3bc-72ad-4164-a414-57b6aa7bd512] AMQP server on 172.18.181.37:5672 is unreachable: <recoverableconnectionerror: error="" unknown="">. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <recoverableconnectionerror: error="" unknown=""> 2020-05-29 14:05:47.810 13263 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: OSError: Server unexpectedly closed connection 2020-05-29 14:05:50.961 13263 WARNING oslo_db.sqlalchemy.engines [req-c08d8d98-c096-4530-a988-1142331a088e b6a49431b1e04ccf8c5d0a9a6320f57d 005b1804ab0b4b3296e69bc2c3724ff0 - f1ceb8de28184e8cb578c8b699cc9fa5 f1ceb8de28184e8cb578c8b699cc9fa5] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.18.181.7' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) 2020-05-29 14:06:04.021 13263 WARNING oslo_db.sqlalchemy.engines [req-c08d8d98-c096-4530-a988-1142331a088e b6a49431b1e04ccf8c5d0a9a6320f57d 005b1804ab0b4b3296e69bc2c3724ff0 - f1ceb8de28184e8cb578c8b699cc9fa5 f1ceb8de28184e8cb578c8b699cc9fa5] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.18.181.7' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8)</recoverableconnectionerror:></recoverableconnectionerror:>

I don't have more this mysql server 172.18.181.7, was old mysql unit. How can I remove it, please? I don't know if can be the root cause.

$ openstack compute service list +-----+------------------+--------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +-----+------------------+--------------------------+----------+---------+-------+----------------------------+ | 1 | nova-conductor | juju-8b2c75-5-lxd-0 | internal | enabled | down | 2020-05-29T14:13:23.000000 | | 19 | nova-scheduler | juju-8b2c75-5-lxd-0 | internal | enabled | down | 2020-05-29T14:13:18.000000 | | 58 | nova-compute | cpu001.cl.openstack.corp | nova | enabled | down | 2020-05-29T14:13:25.000000 | | 61 | nova-compute | cpu005.cl.openstack.corp | nova | enabled | down | 2020-05-29T14:13:25.000000 | | 64 | nova-compute | cpu004.cl.openstack.corp | nova | enabled | down | 2020-05-29T14:13:23.000000 | | 67 | nova-compute | cpu003.cl.openstack.corp | nova | enabled | down | 2020-05-29T14:13:26.000000 | | 70 | nova-compute | cpu002.cl.openstack.corp | nova | enabled | down | 2020-05-29T14:13:24.000000 | | 86 | nova-consoleauth | juju-8b2c75-5-lxd-0 | internal | enabled | down | 2020-05-29T14:13:25.000000 | | 89 | nova-consoleauth | juju-8b2c75-6-lxd-8 | internal | enabled | down | 2020-03-02T20:19:22.000000 | | 92 | nova-conductor | juju-8b2c75-6-lxd-8 | internal | enabled | down | 2020-03-02T20 ... (more)

edit retag flag offensive close merge delete

Comments

The DB connection is defined in nova.conf. Interesting to see that an inaccessible database brings all Nova services down.

Bernd Bausch gravatar imageBernd Bausch ( 2020-05-29 10:10:32 -0500 )edit

Hi, I added more details about mysql connection, mysql units and vip address. So this env was working well, after reboot controller02, this situations started. This mysql unit 172.18.171.7 was removed after issue happened, trying to solve it.

dgolive gravatar imagedgolive ( 2020-05-29 20:36:57 -0500 )edit

Was Nova restarted after the config change? What do you find in the logs when you restart Nova services now - still the erroneous address 172.18.181.7?

If you can't get rid of this address, try a brute force search, e.g. grep -rI 172.18.181.7 /etc /usr /var.

Bernd Bausch gravatar imageBernd Bausch ( 2020-05-29 22:11:47 -0500 )edit

However, there is a problem with RabbitMQ as well:

AMQP server on 172.18.181.37:5672 is unreachable

This is yet another IP address.

Bernd Bausch gravatar imageBernd Bausch ( 2020-05-29 22:13:06 -0500 )edit

By the way, it would be nice of you changed the formatting so that the code becomes readable. The "101010" button does the trick.

Bernd Bausch gravatar imageBernd Bausch ( 2020-05-29 22:16:07 -0500 )edit