Image_Pending_Upload Increases Each Snapshot

asked 2019-05-10 09:46:04 -0500

cprestia gravatar image

updated 2019-05-14 11:03:32 -0500

I take snapshots of the same instances every week. I noticed when an instance is continuously changing, the snapshot starts to take longer. I had an instance grow from being 470GB to 507GB and the backup changed from taking 4 hours to 21 hours. It would take about an hour to two hours longer each week it backed up.

When looking at my logs I realized it was the image_pending_upload that was taking the time. My first backup, when my instance was 470GB on January 1st, took 3 hours in image_pending_upload and 1 hour in image_uploading. My latest backup, when it was 507GB on May 19th, was in image_pending_upload for 20 hours and image_uploading for 1 hour. For comparison, I have another instance that is 530GB and is not in use that is only in image_pending_upload for 50 minutes and image_uploading for an hour and twenty minutes.

The only explanation I have thought of is that the delta between the backing file and the the instance is continuously growing, and each time I take a snapshot it is going through all the changes that have happened since the instance was created.

The solution I came up with was to go to cd /var/lib/nova/instances/, choose my instance id, and run qemu-img commit -f qcow2 disk. This process takes a long time, a little over 24 hours, and requires me to take down my instances whenever my backup time gets to long. After I did this my instance that was taking 21 hours to backup took two and a half hours. Is there a way to prevent image_pending_upload from growing so large?

edit retag flag offensive close merge delete