Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Image_Pending_Upload Unexpectedly Takes Longer When Instances are in Use

Due to production, I take snapshots of instances every week. I noticed when an instance is continuously changing, the snapshot starts to take unexpectedly longer. I had an instance grow from being 470GB to 507GB and the backup changed from taking 4 hours to 21 hours. It would take about an hour to two hours longer each week it backed up.

When looking at my logs I realized it was the image_pending_upload that was taking the time. My first backup, when my instance was 470GB on January 1st, took 3 hours in image_pending_upload and 1 hour in image_uploading. My latest backup, when it was 507GB on May 19th, was in image_pending_upload for 20 hours and image_uploading for 1 hour. For comparison, I have another instance that is 530GB and is not in use that is only in image_pending_upload for 50 minutes and image_uploading for an hour and twenty minutes.

The only explanation I have thought of is that the delta between the backing file and the the instance is continuously growing, and each time I take a snapshot it is going through all the changes that have happened since the instance was created.

The solution I came up with was to go to cd /var/lib/nova/instances/, choose my instance id, and run qemu-img commit -f qcow2 disk. This process takes a long time, a little over 24 hours, and requires me to take down my instances whenever my backup time gets to long. Is there a way to prevent image_pending_upload from growing, such as OpenStack committing the changes during a snapshot?

Image_Pending_Upload Unexpectedly Takes Longer When Instances are in Use

Due to production, I take snapshots of instances every week. I noticed when an instance is continuously changing, the snapshot starts to take unexpectedly longer. I had an instance grow from being 470GB to 507GB and the backup changed from taking 4 hours to 21 hours. It would take about an hour to two hours longer each week it backed up.

When looking at my logs I realized it was the image_pending_upload that was taking the time. My first backup, when my instance was 470GB on January 1st, took 3 hours in image_pending_upload and 1 hour in image_uploading. My latest backup, when it was 507GB on May 19th, was in image_pending_upload for 20 hours and image_uploading for 1 hour. For comparison, I have another instance that is 530GB and is not in use that is only in image_pending_upload for 50 minutes and image_uploading for an hour and twenty minutes.

The only explanation I have thought of is that the delta between the backing file and the the instance is continuously growing, and each time I take a snapshot it is going through all the changes that have happened since the instance was created.

The solution I came up with was to go to cd /var/lib/nova/instances/, choose my instance id, and run qemu-img commit -f qcow2 disk. This process takes a long time, a little over 24 hours, and requires me to take down my instances whenever my backup time gets to long. Is there a way to prevent image_pending_upload from growing, such as OpenStack committing the changes during a snapshot?

Image_Pending_Upload Unexpectedly Takes Longer When Instances are in UseIncreases Each Snapshot

Due to production, I take snapshots of the same instances every week. I noticed when an instance is continuously changing, the snapshot starts to take unexpectedly longer. I had an instance grow from being 470GB to 507GB and the backup changed from taking 4 hours to 21 hours. It would take about an hour to two hours longer each week it backed up.

When looking at my logs I realized it was the image_pending_upload that was taking the time. My first backup, when my instance was 470GB on January 1st, took 3 hours in image_pending_upload and 1 hour in image_uploading. My latest backup, when it was 507GB on May 19th, was in image_pending_upload for 20 hours and image_uploading for 1 hour. For comparison, I have another instance that is 530GB and is not in use that is only in image_pending_upload for 50 minutes and image_uploading for an hour and twenty minutes.

The only explanation I have thought of is that the delta between the backing file and the the instance is continuously growing, and each time I take a snapshot it is going through all the changes that have happened since the instance was created.

The solution I came up with was to go to cd /var/lib/nova/instances/, choose my instance id, and run qemu-img commit -f qcow2 disk. This process takes a long time, a little over 24 hours, and requires me to take down my instances whenever my backup time gets to long. After I did this my instance that was taking 21 hours to backup took two and a half hours. Is there a way to prevent image_pending_upload from growing, such as OpenStack committing the changes during a snapshot?growing so large?

Image_Pending_Upload Increases Each Snapshot

I take snapshots of the same instances every week. I noticed when an instance is continuously changing, the snapshot starts to take longer. I had an instance grow from being 470GB to 507GB and the backup changed from taking 4 hours to 21 hours. It would take about an hour to two hours longer each week it backed up.

When looking at my logs I realized it was the image_pending_upload that was taking the time. My first backup, when my instance was 470GB on January 1st, took 3 hours in image_pending_upload and 1 hour in image_uploading. My latest backup, when it was 507GB on May 19th, was in image_pending_upload for 20 hours and image_uploading for 1 hour. For comparison, I have another instance that is 530GB and is not in use that is only in image_pending_upload for 50 minutes and image_uploading for an hour and twenty minutes.

The only explanation I have thought of is that the delta between the backing file and the the instance is continuously growing, and each time I take a snapshot it is going through all the changes that have happened since the instance was created.

The solution I came up with was to go to cd /var/lib/nova/instances/, choose my instance id, and run qemu-img commit -f qcow2 disk. This process takes a long time, a little over 24 hours, and requires me to take down my instances whenever my backup time gets to long. After I did this my instance that was taking 21 hours to backup took two and a half hours. Is there any way to commit the changes when a way to prevent image_pending_upload from growing snapshot occurs so large?image_pending_upload does not take as long?