Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Launching multiple fails cause of VolumeNotCreated: ... did not finish being created even after we waited 190 seconds or 61 attempts.

Hi,

got a fresh openstack juno installation on ubuntu 14.04 with a ceph rbd storage backend. Everything works fine expect when i try to launch multiple instances using 'Boot from image (creates a new volume', tested 20 instances with volume size of 20Gig, image in format raw (2,2Gig).

The nova.log on the compute nodes shows

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1825, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] do_check_attach=do_check_attach) + 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 407, in attach_block_devices 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] map(_log_and_attach, block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 405, in _log_and_attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] bdm.attach(attach_args, *attach_kwargs) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 333, in attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] wait_func(context, vol['id']) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1269, in _await_block_device_map_created 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] attempts=attempts) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] VolumeNotCreated: Volume b6454952-ddad-426d-aff2-f600fedc9748 did not finish being created even after we waited 190 seconds or 61 attempts.

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.488 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Failure prepping block device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2224, in _build_resources 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1853, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] raise exception.InvalidBDM() 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] InvalidBDM: Block Device Mapping is Invalid.

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.548 24632 INFO nova.scheduler.client.report [-] Compute_service record updated for ('node01', 'node01.wcloud.uni-koblenz.de') 2015-01-09 17:34:40.549 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Build of instance 719285ff-ef90-41f3-9a0e-2dfa97487b17 aborted: Failure prepping block device. 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2036, in _do_build_and_run_instance 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] filter_properties) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2135, in _build_and_run_instance 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 'create.error', fault=e) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__ 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] six.reraise(self.type_, self.value, self.tb) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2108, in _build_and_run_instance 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] block_device_mapping) as resources: 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] return self.gen.next() 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2246, in _build_resources 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] reason=msg) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] BuildAbortException: Build of instance 719285ff-ef90-41f3-9a0e-2dfa97487b17 aborted: Failure prepping block device. 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17]

For me it looks like that nova-compute is just not waiting long enough for the volumes to be created. Later, after all of the instance creations fail, the volumes are finally created properly with 'available' state and can be attached to running instances. Is there a way to extend the timeout of nova waiting for volumes to be created??

Single instances or smaller numbers of instances can be created without running in these issues.

Stefan

Launching multiple fails cause of VolumeNotCreated: ... did not finish being created even after we waited 190 seconds or 61 attempts.

Hi,

got a fresh openstack juno installation on ubuntu 14.04 with a ceph rbd storage backend. Everything works fine expect when i try to launch multiple instances using 'Boot from image (creates a new volume', tested 20 instances with volume size of 20Gig, image in format raw (2,2Gig).

The nova.log on the compute nodes shows

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1825, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] do_check_attach=do_check_attach) + 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 407, in attach_block_devices 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] map(_log_and_attach, block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 405, in _log_and_attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] bdm.attach(attach_args, *attach_kwargs) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 333, in attach 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] wait_func(context, vol['id']) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1269, in _await_block_device_map_created 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] attempts=attempts) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] VolumeNotCreated: Volume b6454952-ddad-426d-aff2-f600fedc9748 did not finish being created even after we waited 190 seconds or 61 attempts.

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.488 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Failure prepping block device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2224, in _build_resources 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] block_device_mapping) 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1853, in _prep_block_device 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] raise exception.InvalidBDM() 2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] InvalidBDM: Block Device Mapping is Invalid.

2015-01-09 17:34:40.488 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 2015-01-09 17:34:40.548 24632 INFO nova.scheduler.client.report [-] Compute_service record updated for ('node01', 'node01.wcloud.uni-koblenz.de') 2015-01-09 17:34:40.549 24632 ERROR nova.compute.manager [-] [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Build of instance 719285ff-ef90-41f3-9a0e-2dfa97487b17 aborted: Failure prepping block device. 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] Traceback (most recent call last): 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2036, in _do_build_and_run_instance 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] filter_properties) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2135, in _build_and_run_instance 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] 'create.error', fault=e) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__ 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] six.reraise(self.type_, self.value, self.tb) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2108, in _build_and_run_instance 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] block_device_mapping) as resources: 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] return self.gen.next() 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2246, in _build_resources 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] reason=msg) 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17] BuildAbortException: Build of instance 719285ff-ef90-41f3-9a0e-2dfa97487b17 aborted: Failure prepping block device. 2015-01-09 17:34:40.549 24632 TRACE nova.compute.manager [instance: 719285ff-ef90-41f3-9a0e-2dfa97487b17]

For me it looks like that nova-compute is just not waiting long enough for the volumes to be created. Later, after all of the instance creations fail, the volumes are finally created properly with 'available' state and can be attached to running instances. Is there a way to extend the timeout of nova waiting for volumes to be created??

Single instances or smaller numbers of instances can be created without running in these issues.

Same behavior, when i try to launch a single (windows) instance with a relatively big (15Gb) image file using 'Boot from image (creates a new volume'. The instance building times out and returns the same error. Then the volume creation (coying the image) finally finishes (after the error from nova). The created volume is fine and a vm can be booted from it.

Stefan