Ask Your Question
0

Launching multiple fails cause of VolumeNotCreated: ... did not finish being created even after we waited 190 seconds or 61 attempts.

asked 2015-01-09 11:31:18 -0600

schegi gravatar image

updated 2015-01-20 09:54:25 -0600

Hi,

got a fresh openstack juno installation on ubuntu 14.04 with a ceph rbd storage backend. Everything works fine expect when i try to launch multiple instances using 'Boot from image (creates a new volume', tested 20 instances with volume size of 20Gig, image in format raw (2,2Gig).

The nova.log on the compute nodes shows

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1825, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] do_check_attach=do_check_attach) + 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 407, in attach_block_devices 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] map(_log_and_attach, block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 405, in _log_and_attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] bdm.attach(attach_args, *attach_kwargs) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 333, in attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] wait_func(context, vol['id']) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1269, in _await_block_device_map_created 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] attempts=attempts) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] VolumeNotCreated: Volume b6454952-ddad-426d-aff2-f600fedc9748 did not finish being created even after we waited 190 seconds or 61 attempts.

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.488 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Failure prepping block device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2224, in _build_resources 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1853, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] raise exception.InvalidBDM() 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] InvalidBDM: Block Device Mapping is Invalid.

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.548 24632 INFO nova.scheduler.client.report [-] Compute_service record updated for ('node01', 'node01.wcloud.uni-koblenz.de') 2015-01-09 17:34:40.549 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Build of instance 719285ff-ef90-41f3-9a0e-2dfa97487b17 ...

(more)
edit retag flag offensive close merge delete

Comments

Volume b6454952-ddad-426d-aff2-f600fedc9748 not created ? Or check the volume's state if "available".

Eric Xie gravatar imageEric Xie ( 2015-01-09 17:43:13 -0600 )edit

Volume is finally created, but too late. The instance launch fails cause of the timeout 'VolumeNotCreated: Volume b6454952-ddad-426d-aff2-f600fedc9748 did not finish being created even after we waited 190 seconds or 61 attempts.'. Volume creation itself is not the problem, but instance launch.

schegi gravatar imageschegi ( 2015-01-10 09:42:58 -0600 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2015-01-21 06:36:20 -0600

schegi gravatar image

updated 2015-01-21 11:50:01 -0600

The issue was somehow my fault. While using a rbd storage backend it is highly recommended to set the

show_image_direct_url = True

option in the glance-api.conf. With this option the image is copied directly by the rbd store and finally the volume creation/image copy only takes a couple of seconds. Now the instantiation of multiple instances and of very big images is no longer a problem.

Greets Stefan

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-01-09 11:31:18 -0600

Seen: 1,733 times

Last updated: Jan 21 '15