Ask Your Question
0

Cinder deadlock - Unable to create attachment

asked 2020-07-28 17:08:15 -0500

tonda gravatar image

updated 2020-07-29 09:52:48 -0500

Good evening.

When creating multiple servers in stack I sometimes get following error:

...
2020-07-28 16:29:57Z [compute_test.compute-test6]: CREATE_IN_PROGRESS  state changed
...
020-07-28 16:30:11Z [compute_test]: CREATE_FAILED  Resource CREATE failed: BadRequest: resources.compute-test6.resources.server_with_volume: Block Device Mapping is Invalid: failed to get volume 08293421-3a68-4d76-956f-a9145c9cfd1d. (HTTP 400) (Request-ID: req-76b18c65-5c6d-44c9-a814-8e893be24a0a)

Cinder log:

cinder-volume.log:2020-07-28 16:30:04.601 3543 INFO cinder.volume.flows.manager.create_volume [req-068cce26-28f3-4723-9944-8a984d687091 d3679214a5cbb7b0efea4e21086e673b6184d1d1d1586cbb612edd808b600eee d7ab62d3c3f1415ab99b2d2cb9a8de19 - 7b0f68399f984a6ba59344179283d639 7b0f68399f984a6ba59344179283d639] Volume volume-08293421-3a68-4d76-956f-a9145c9cfd1d (08293421-3a68-4d76-956f-a9145c9cfd1d): created successfully


cinder-api.log:2020-07-28 16:30:11.229 6259 ERROR cinder.api.v3.attachments [req-38b46585-7efc-4b28-b80b-95f180d635bb d3679214a5cbb7b0efea4e21086e673b6184d1d1d1586cbb612edd808b600eee d7ab62d3c3f1415ab99b2d2cb9a8de19 - 7b0f68399f984a6ba59344179283d639 7b0f68399f984a6ba59344179283d639] Unable to create attachment for volume.: oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') [SQL: 'INSERT INTO volume_attachment (created_at, updated_at, deleted_at, deleted, id, volume_id, instance_uuid, attached_host, mountpoint, attach_time, detach_time, attach_status, attach_mode, connection_info, connector) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(id)s, %(volume_id)s, %(instance_uuid)s, %(attached_host)s, %(mountpoint)s, %(attach_time)s, %(detach_time)s, %(attach_status)s, %(attach_mode)s, %(connection_info)s, %(connector)s)'] [parameters: {'created_at': datetime.datetime(2020, 7, 28, 16, 30, 9, 745185), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'id': 'a18ea483-e17e-4ede-8d65-98fc64e9b3d4', 'volume_id': '08293421-3a68-4d76-956f-a9145c9cfd1d', 'instance_uuid': '53ca04b1-b31a-4608-ba59-f2ef4e84c0fc', 'attached_host': None, 'mountpoint': None, 'attach_time': None, 'detach_time': None, 'attach_status': 'reserved', 'attach_mode': None, 'connection_info': None, 'connector': None}] (Background on this error at: http://sqlalche.me/e/2j85)
cinder-api.log:2020-07-28 18:30:11.231 6259 INFO cinder.api.openstack.wsgi [req-38b46585-7efc-4b28-b80b-95f180d635bb d3679214a5cbb7b0efea4e21086e673b6184d1d1d1586cbb612edd808b600eee d7ab62d3c3f1415ab99b2d2cb9a8de19 - 7b0f68399f984a6ba59344179283d639 7b0f68399f984a6ba59344179283d639] HTTP exception thrown: Unable to create attachment for volume.

Nova log:

nova-api.log:2020-07-28 16:30:11.238 13186 ERROR nova.volume.cinder [req-76b18c65-5c6d-44c9-a814-8e893be24a0a d3679214a5cbb7b0efea4e21086e673b6184d1d1d1586cbb612edd808b600eee d7ab62d3c3f1415ab99b2d2cb9a8de19 - 7b0f68399f984a6ba59344179283d639 7b0f68399f984a6ba59344179283d639] [instance: 53ca04b1-b31a-4608-ba59-f2ef4e84c0fc] Create attachment failed for volume 08293421-3a68-4d76-956f-a9145c9cfd1d. Error: Unable to create attachment for volume.

Tried increasing block_device_creation_timeout=60 from 10 in nova.conf on compute nodes and nova hosts, no change.

Also using: block_device_allocate_retries=600 block_device_allocate_retries_interval=10

Versions - Stein:

  • nova: 19.0.1
  • heat: 12.0.0
  • cinder: 14.0.0

Volume backend: rbd ceph Nova backend: libvirt Database: single node master MySQL with replication to 1 slave

Issue is not specific to any compute node, random

edit retag flag offensive close merge delete

Comments

Check the nova-compute logs, too.

Bernd Bausch gravatar imageBernd Bausch ( 2020-07-28 17:37:39 -0500 )edit

Sorry, edit with cinder deadlock didnt make it in original post, fixed.

Nova-compute log on hypervisor shows nothing interesting, even with debug mode.

tonda gravatar imagetonda ( 2020-07-29 02:31:48 -0500 )edit

for some reason im not able to edit the post again: more cinder info https://pastebin.com/Z7ZL1hBP

tonda gravatar imagetonda ( 2020-07-29 03:24:42 -0500 )edit

This sounds like a database issue to me, maybe caused by version mismatch? Does a cinder-manage db sync work?

eblock gravatar imageeblock ( 2020-07-30 01:00:24 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2020-07-30 09:38:00 -0500

zaneb gravatar image

This is definitely a bug in Cinder - if the DB transaction deadlocks it needs to be retried. It's possible this could have been fixed or at least reported already, so I suggest you look around and open a bug report if none already exists.

edit flag offensive delete link more

Comments

Thanks for info, didnt find any suitable help in existing bug reports, created a new one https://bugs.launchpad.net/cinder/+bug/1889667 (https://bugs.launchpad.net/cinder/+bu...)

tonda gravatar imagetonda ( 2020-07-30 12:05:58 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

Stats

Asked: 2020-07-28 12:30:37 -0500

Seen: 0 times

Last updated: Jul 30