nova-conductor repeatedly causing deadlock on mysql

asked 2017-04-27 11:05:43 -0600

niv gravatar image

hi, i have mariadb with galera cluster (3 nodes) (mysql Ver 15.1) running on ubuntu 16.04. when trying to create a new instance, i get a error message on nova-conductor.log that there is a deadlock on mysql. the problem happens on nova.services table. about 120 processes trying to update the 'updated_at' column, causing deadlock the command 'SHOW OPEN TABLES WHERE In_use > 0', returns that nova.services is open around 125 times (it changes). the command 'SHOW processlist' returns a lot of processes from nova, trying to update the same table (services), all in QUERY state. when i turn down nova-conductor, the use counter goes back to 0, but when i turn it back on, it create deadlocks straight away! It happened more than once.

logs :

/var/log/nova/nova-scheduler.log - 2017-04-27 09:28:44 42798 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 9.06 sec ( this line appeared again and again)

/var/log/nova/nova-conductor.log - 2017-04-27 09:28:50 42542 ERROR oslo_db.api [-] DB exceeded retry limit. ... (stack trace) 2017-04-27 09:28:50 42542 ERROR oslo_db.api DBDeadlock: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded, try restarting transaction') [SQL: u'UPDATE services SET updated_at=%(updated_at)s, report_count=%(report_count)s WHERE services.id = %(services_id)s'] [parameters: {'services_id':73, 'updated_at' : datetime.datetime(2017,4,27,13,27,59,586164), 'report_count': 22893}]

2017-04-27 09:28:50 42542 ERROR nova.servicegroups.drivers.db [-] Unexpected error while reporting service status ... (stack trace) 2017-04-27 09:28:50 42542 ERROR nova.servicegroups.drivers.db DBDeadlock: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded, try restarting transaction') [SQL: u'UPDATE services SET updated_at=%(updated_at)s, report_count=%(report_count)s WHERE services.id = %(services_id)s'] [parameters: {'services_id':73, 'updated_at' : datetime.datetime(2017,4,27,13,27,59,586164), 'report_count': 22893}]

*NOTE: the same error repeats once every few seconds.

edit retag flag offensive close merge delete