Launching multiple fails cause of VolumeNotCreated: ... did not finish being created even after we waited 190 seconds or 61 attempts.
Hi,
got a fresh openstack juno installation on ubuntu 14.04 with a ceph rbd storage backend. Everything works fine expect when i try to launch multiple instances using 'Boot from image (creates a new volume', tested 20 instances with volume size of 20Gig, image in format raw (2,2Gig).
The nova.log on the compute nodes shows
2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1825, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] do_check_attach=do_check_attach) + 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 407, in attach_block_devices 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] map(_log_and_attach, block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 405, in _log_and_attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] bdm.attach(attach_args, *attach_kwargs) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 333, in attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] wait_func(context, vol['id']) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1269, in _await_block_device_map_created 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] attempts=attempts) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] VolumeNotCreated: Volume b6454952-ddad-426d-aff2-f600fedc9748 did not finish being created even after we waited 190 seconds or 61 attempts.
2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.488 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Failure prepping block device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2224, in _build_resources 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1853, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] raise exception.InvalidBDM() 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] InvalidBDM: Block Device Mapping is Invalid.
2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.548 24632 INFO nova.scheduler.client.report [-] Compute_service record updated for ('node01', 'node01.wcloud.uni-koblenz.de') 2015-01-09 17:34:40.549 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Build of instance 719285ff-ef90-41f3-9a0e-2dfa97487b17 ...
Volume b6454952-ddad-426d-aff2-f600fedc9748 not created ? Or check the volume's state if "available".
Volume is finally created, but too late. The instance launch fails cause of the timeout 'VolumeNotCreated: Volume b6454952-ddad-426d-aff2-f600fedc9748 did not finish being created even after we waited 190 seconds or 61 attempts.'. Volume creation itself is not the problem, but instance launch.