Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Error writing to /var/lib/nova/instances/_base/ - how to avoid local file transaction?

Hi,

I am using Ceph for RBD storage and we are hosting hypervisors with 16GB of flash memory.

When I import an image which is saved in an isolated pool on the same cluster, my hypervisor node tries to download it before re-importing (see trace below). How can I modify this behavior?

2016-04-19 22:46:05.041 2747 ERROR nova.image.glance [req-7f1512f5-4cc7-44fe-bb90-a1527e3a341b 051f7eb0c4df40dda84a69d40ee86a48 3c297aff8cb44e618fb88356a2dd836b - - -] Error writing to /var/lib/nova/instances/_base/1cb800df157f685e900195a14c05d2d4decc2bc4.part: [Errno 28] No space left on device
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [req-7f1512f5-4cc7-44fe-bb90-a1527e3a341b 051f7eb0c4df40dda84a69d40ee86a48 3c297aff8cb44e618fb88356a2dd836b - - -] [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7] Instance failed to spawn
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7] Traceback (most recent call last):
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     yield resources
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     block_device_info=block_device_info)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2761, in spawn
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     admin_pass=admin_password)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3179, in _create_image
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     instance, size, fallback_from_host)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6739, in _try_fetch_image_cache
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     size=size)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 251, in cache
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     *args, **kwargs)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 874, in create_image
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     prepare_template(target=base, max_size=size, *args, **kwargs)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     return f(*args, **kwargs)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in fetch_func_sync
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     fetch_func(target=target, *args, **kwargs)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3173, in clone_fallback_to_fetch
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     libvirt_utils.fetch_image(*args, **kwargs)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 429, in fetch_image
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     max_size=max_size)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 115, in fetch_to_raw
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     max_size=max_size)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 105, in fetch
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     IMAGE_API.download(context, image_href, dest_path=path)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 182, in download
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     dst_path=dest_path)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 446, in download
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     {'path': dst_path, 'exception': ex})
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     self.force_reraise()
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     six.reraise(self.type_, self.value, self.tb)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 433, in download
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]     data.write(chunk)
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7] IOError: [Errno 28] No space left on device
2016-04-19 22:46:05.730 2747 ERROR nova.compute.manager [instance: 65bf98e0-a199-4493-82bb-3fee877d6ce7]