instance snapshot fails to complete

asked 2020-02-27 16:57:41 -0500

toddleish gravatar image

I have several instances on one compute node. I can successfully "Create Snapshot" for all but one instance. The instance that fails to complete is much larger than the others, but at 99GB, there should still be plenty of disk space to create the temporary snapshot file. The temp snapshot file never gets bigger than 176KB.

The compute node is using a local NVMe drive for instance storage.

I have watched the local nova log and the glance log, but there are no errors. The process never gets past

2020-02-27 14:39:16.166 31638 INFO nova.virt.libvirt.driver [req-45708b6c-d892-4a87-ab82-2e68bfb8deb7 d63af2d376de4f68af54f1afdea41f8c d82194f0fc3a440faf5e73aae5337b61 - default default] [instance: 4a4fed08-ab95-4291-9841-c5a00a8cd7f5] Beginning cold snapshot process

I get the same result whether I run this via the Horizon interface or via the CLI. Is there anywhere else I can look to see why this snapshot is not processing? I am running OpenStack Stein (RDO Packstack with 6 compute nodes)

edit retag flag offensive close merge delete

Comments

Could you enable debug logs and run that again?

eblock gravatar imageeblock ( 2020-02-28 01:02:33 -0500 )edit

Hi... Yes, that was run with debug logs on, and that was the only thing I saw related to the snapshot process -- for hours... Should I post the entire log?

toddleish gravatar imagetoddleish ( 2020-02-28 07:03:27 -0500 )edit

Nova instructs glance to take the snapshot, did you also take a look into the control node(s)? Maybe glance runs out of space or something different. Try also with debug logs in glancen nova on control node.

eblock gravatar imageeblock ( 2020-02-28 07:13:24 -0500 )edit

Yes, I had done debug mode on those control node logs as well, but no issues could be seen... Could you step me through the process Snapshot uses to create/move/store the snapshot?

toddleish gravatar imagetoddleish ( 2020-02-28 10:14:36 -0500 )edit

What is your storage backend, especially for glance?

eblock gravatar imageeblock ( 2020-02-28 13:01:02 -0500 )edit