nova-api error every 5 minutes

asked 2020-04-04 13:06:20 -0500

tien86 gravatar image

Dear,

I have 2 problems hope people can take a look for this experience. I try to find whole days and still stuck,

I install 3 node controller + 1 node lb haproxy a, I check my nova-api.logs its happen every 5 mins ( 300s ) :

2020-04-04 03:29:29.656 2760 ERROR oslo_db.sqlalchemy.engines [req-1f87e24a-3dc6-4c61-84a3-f729f424d21b 5d1d17dde47a4cb38cd638b194dda114 fbd1a24058684494a1ecfa8774abeb60 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) ...

2020-04-04 03:34:27.214 2757 ERROR oslo_db.sqlalchemy.engines [req-68966151-5d66-4ad4-893a-68931664b3b4 5d1d17dde47a4cb38cd638b194dda114 fbd1a24058684494a1ecfa8774abeb60 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)

... 2020-04-04 03:39:28.293 2759 ERROR oslo_db.sqlalchemy.engines [req-ad2e5eb3-9674-457b-970e-743612b7d98a 5d1d17dde47a4cb38cd638b194dda114 fbd1a24058684494a1ecfa8774abeb60 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)

I guess there some timeout in haproxy or nova or mariadb.

mariadb +---------------------------------------+----------+ | Variable_name | Value | +---------------------------------------+----------+ | connect_timeout | 1800 | | deadlock_timeout_long | 50000000 | | deadlock_timeout_short | 10000 | | delayed_insert_timeout | 300 | | idle_readonly_transaction_timeout | 0 | | idle_transaction_timeout | 0 | | idle_write_transaction_timeout | 0 | | innodb_flush_log_at_timeout | 1 | | innodb_lock_wait_timeout | 5 | | innodb_rollback_on_timeout | OFF | | interactive_timeout | 28800 | | lock_wait_timeout | 86400 | | net_read_timeout | 30 | | net_write_timeout | 60 | | rpl_semi_sync_master_timeout | 10000 | | rpl_semi_sync_slave_kill_conn_timeout | 5 | | slave_net_timeout | 60 | | thread_pool_idle_timeout | 60 | | wait_timeout | 28800 | +---------------------------------------+----------+

haproxy defaults log global mode tcp maxconn 10000 timeout connect 2s timeout client 10s timeout server 10s

listen galera_cluster *:3306 timeout client 90s timeout server 90s

listen neutron_api *:9696 balance source option tcpka option httpchk option tcplog server control1 10.1.17.101:9696 check inter 2000 rise 2 fall 5 server control2 10.1.17.102:9696 check inter 2000 rise 2 fall 5 server control3 10.1.17.103:9696 check inter 2000 rise 2 fall 5

nova.conf all defaults, I find 2 parameters relate to database however timeout still higher

conn_pool_ttl Type integer Default 1200 The time-to-live in sec of idle connections in the pool

From https://docs.openstack.org/nova/latest/configuration/config.html

connection_recycle_time Default 3600 From https://docs.openstack.org/oslo.db/latest/reference/opts.html

edit retag flag offensive close merge delete