snapshot not working when using ceph [closed]

asked 2015-01-15 17:55:01 -0500

capsali gravatar image

updated 2015-01-21 07:22:01 -0500

Hi,

I am having some trouble taking snapshots in juno! I am using ceph as a backend for glance cinder and ephemeral.

My setup is : 2 controller nodes, 3 compute nodes and 2 storage nodes running in HA.

Any snapshot i take from a VM that is 40Gb or less in size, the process completes but it takes a long time.

Above the 40Gb limit the snapshot gets stuck at saving indefinetly.

The thing is if i look at ceph df i can see that the pool in which glance stores images is starting to grow and stops at around the vm's size. But the new image doesn't get in active state and gets stuck in saving state.

I don't know if it is a coincidence or not but i have around 40Gb free space on all compute nodes and i know that mometarily juno does not support rbd snapshotting and relies on qemu for snapshot so theoreticly it should download the image on the compute node that the VM resides on and then upload it into glance.

But if i watch the hdd utilization on the compute node when the snapshotting is taking place on any size of the instance, there is max 10Gb downloaded locally, so the copmute nodes hdd doesnt get full on any size of the VM.

So my conclusion is that the snapshot is not downloaded entirely on local storage but uploaded directly into glance

Could this be a glance glitch or something? I haven't tried to reset a snapshot state from saving to active. Can we do that with glance like a nova reset command?

Any ideas on how to fix this?

Also there is nothing in the logs that is relevant to this situation, neither in glance logs nor in nova logs!

UPDATE:

It seems that this is a dashboard related problem only. If i use command line like nova image-create the snapshot is created and appears as active under images tab!

Any idea why under horizon instance snapshot gets stuck at saving, even thouhg the snapshot is uploaded into glance on ceph?

I'm guessing status isn't being refreshed after the snapshot get's uploaded in glance!

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by capsali
close date 2015-06-24 06:55:23.787473

Comments

I'm having the same issue on ubuntu/trusty w/ ceph backend. cinder vol snaps work just fine. instance snaps take a VERY long time . Not sure why. Nothing stands out in the logs.

neerbeer gravatar imageneerbeer ( 2015-02-09 13:41:18 -0500 )edit

This is because when snapshotting an instance it doesnt use ceph snapshot but instead it uses qemu snapshot, so it first downloads a copy of the instance on local storage, than it uploads it into ceph. It will be fixed in kilo release.

capsali gravatar imagecapsali ( 2015-02-10 05:43:23 -0500 )edit

my problem is that when i take a snapshot from dashboard it never completes, it's stuck in saving. When i take the snapshot from the terminal it works.

capsali gravatar imagecapsali ( 2015-02-10 05:44:14 -0500 )edit

I am also having the same issue. I have to go for a hit and trial in most of the cases, and to make it worst today I tried creating a snaphsot using command line, still it remains stuck in saving status. My setup is using Kilo.

Ashish Chandra gravatar imageAshish Chandra ( 2016-12-07 03:54:31 -0500 )edit