Ask Your Question
2

Why is juno ceph using "qemu-img convert" for instance snapshots?

asked 2015-02-09 14:38:33 -0500

neerbeer gravatar image

updated 2015-02-09 14:38:56 -0500

I'm running juno w/ ceph(giant) . I can boot VMs quickly w/ ceph backend. I can create and snapshot volumes just fine.

When I goto take a snapshot of a running instance I see "qemu-img convert" in my log nova-compute.log on the compute node.

DEBUG nova.openstack.common.processutils [req-0d208ea3-bfa1-4209-96ef-6cf16a50f3ce None] Running cmd (subprocess): **qemu-img convert -O raw rbd:vms/260bf377-ce94-48e6-88de-a15c246b0019_disk:id=cinder:conf=/etc/ceph/ceph.conf /var/lib/nova/instances/snapshots/tmpz1T2nq/91ef5aa7338c40679a88b9d93ab1af09** execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:161

Not sure what is suppossed to happen but I would think that snapshots of instances would be very fast w/ ceph. Currently snapshots are completing but they are very,very slow .

The relevant bits from my nova.conf on my compute nodes:

cat /etc/nova/nova.conf
.
.

[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user =  cinder
rbd_secret_uuid = xxx-xxx-xxx
.
.

What am I doing wrong ?

edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted
2

answered 2015-02-17 14:51:34 -0500

neerbeer gravatar image

updated 2015-02-17 20:28:48 -0500

I found the correct answer to my question by going through the code. And the quick answer is that ceph/rbd based snapshots are not supported in Juno. Longer explanation, which may be rather moot but helpful follows:

The overall process is that snapshots are taken and are stored in glance . I didn't expect snapshots to be stored in glance which was part of my misunderstanding.

Further, Breaking this down into two parts a) taking snapshots and b) storing the snapshot(ed) image into glance.

A) Taking shapshots:
When snapshots are taken of an rbd image the image_format is set to 'raw'.

/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py
# NOTE(bfilippov): save lvm and rbd as raw
        if image_format == 'lvm' or image_format == 'rbd':
            image_format = 'raw'

Snapshots are, according to the code, extracted and converted for preparation for upload into glance. This process converts the image to 'raw' format using 'qemu-img convert' in /usr/lib/python2.7/dist-packages/nova/virt/images.py which answers my original question.

B) Storing snapshots into glance:
Once the image is converted it's uploaded into glance. This occurs by coping the image across the network , from the compute node's local filesytem into glance-api ... further adding to the total time before the snapshot is ready for use.

The behavior we want w/ ceph backed openstack is to have the snapshot be done via rbd and registered w/ glance. That way there's no need for conversion and no need to upload anything across the network . Snapshots and image registration instead of image uploads into glance should take brief seconds.

Hopefully, this will be fixed in kilo.

edit flag offensive delete link more

Comments

there is a bug for this: https://bugs.launchpad.net/nova/+bug/... and a draft blueprint: https://blueprints.launchpad.net/nova... currently being adressed in review: https://review.openstack.org/#/c/125963

jproulx gravatar imagejproulx ( 2015-03-25 14:11:36 -0500 )edit
1

answered 2015-02-10 02:35:03 -0500

simon gravatar image

The explanation sounds plausible, but this particular implementation looks less than ideal. In particular, the image seems to be converted to a local filesystem, which is bound to run into space issues. Couldn't qemu-img write it to RBD directly? Or, even better, shouldn't "rbd flatten" be used?

edit flag offensive delete link more
0

answered 2015-02-09 22:49:35 -0500

Instance are provisioned using linked clones. That means only the diff is written to your images disk and the reads are done from base disk. When you create a snapshot, the diff and base disk must be merged to single monolithic disk before import in to glance as an image.

This is the normal process. If only the diff is pushed to glance the snapshot will be useless and not bootable.

edit flag offensive delete link more

Comments

Plausible? This is 100% how its implemented!

Where are you instances running (instances_path in nova.conf) ?

You can also set the location of the snapshot directory to be rbd via noa.conf.

snapshots_directory=$instances_path/snapshots
sfcloudman gravatar imagesfcloudman ( 2015-02-10 14:15:00 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-02-09 14:38:33 -0500

Seen: 1,736 times

Last updated: Feb 17 '15