# Why is juno ceph using "qemu-img convert" for instance snapshots?

I'm running juno w/ ceph(giant) . I can boot VMs quickly w/ ceph backend. I can create and snapshot volumes just fine.

When I goto take a snapshot of a running instance I see "qemu-img convert" in my log nova-compute.log on the compute node.

DEBUG nova.openstack.common.processutils [req-0d208ea3-bfa1-4209-96ef-6cf16a50f3ce None] Running cmd (subprocess): **qemu-img convert -O raw rbd:vms/260bf377-ce94-48e6-88de-a15c246b0019_disk:id=cinder:conf=/etc/ceph/ceph.conf /var/lib/nova/instances/snapshots/tmpz1T2nq/91ef5aa7338c40679a88b9d93ab1af09** execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:161


Not sure what is suppossed to happen but I would think that snapshots of instances would be very fast w/ ceph. Currently snapshots are completing but they are very,very slow .

The relevant bits from my nova.conf on my compute nodes:

cat /etc/nova/nova.conf
.
.

[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user =  cinder
rbd_secret_uuid = xxx-xxx-xxx
.
.


What am I doing wrong ?

edit retag close merge delete

Sort by » oldest newest most voted

I found the correct answer to my question by going through the code. And the quick answer is that ceph/rbd based snapshots are not supported in Juno. Longer explanation, which may be rather moot but helpful follows:

The overall process is that snapshots are taken and are stored in glance . I didn't expect snapshots to be stored in glance which was part of my misunderstanding.

Further, Breaking this down into two parts a) taking snapshots and b) storing the snapshot(ed) image into glance.

A) Taking shapshots:
When snapshots are taken of an rbd image the image_format is set to 'raw'.

/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py
# NOTE(bfilippov): save lvm and rbd as raw
if image_format == 'lvm' or image_format == 'rbd':
image_format = 'raw'


Snapshots are, according to the code, extracted and converted for preparation for upload into glance. This process converts the image to 'raw' format using 'qemu-img convert' in /usr/lib/python2.7/dist-packages/nova/virt/images.py which answers my original question.

B) Storing snapshots into glance:
Once the image is converted it's uploaded into glance. This occurs by coping the image across the network , from the compute node's local filesytem into glance-api ... further adding to the total time before the snapshot is ready for use.

The behavior we want w/ ceph backed openstack is to have the snapshot be done via rbd and registered w/ glance. That way there's no need for conversion and no need to upload anything across the network . Snapshots and image registration instead of image uploads into glance should take brief seconds.

Hopefully, this will be fixed in kilo.

more

there is a bug for this: https://bugs.launchpad.net/nova/+bug/... and a draft blueprint: https://blueprints.launchpad.net/nova... currently being adressed in review: https://review.openstack.org/#/c/125963

( 2015-03-25 14:11:36 -0600 )edit

The explanation sounds plausible, but this particular implementation looks less than ideal. In particular, the image seems to be converted to a local filesystem, which is bound to run into space issues. Couldn't qemu-img write it to RBD directly? Or, even better, shouldn't "rbd flatten" be used?

more

Instance are provisioned using linked clones. That means only the diff is written to your images disk and the reads are done from base disk. When you create a snapshot, the diff and base disk must be merged to single monolithic disk before import in to glance as an image.

This is the normal process. If only the diff is pushed to glance the snapshot will be useless and not bootable.

more

Plausible? This is 100% how its implemented!

Where are you instances running (instances_path in nova.conf) ?

You can also set the location of the snapshot directory to be rbd via noa.conf.

snapshots_directory=\$instances_path/snapshots

( 2015-02-10 14:15:00 -0600 )edit