VM won't start because image has been deleted

asked 2020-07-17 17:06:51 -0600

bnordgren gravatar image

I'm trying to start a VM which has been shutdown for a long time. It may have been off long enough that we're now running a different version of openstack than when it was last running.

When I try to start it, I get:

ImageNotFound: Image cd2bd82f-b9c5-4905-81c6-84e1c946e4e6 could not be found.

Checking:

root@compute-1:~# openstack image show cd2bd82f-b9c5-4905-81c6-84e1c946e4e6                    
Could not find resource cd2bd82f-b9c5-4905-81c6-84e1c946e4e6

And:

root@compute-1:/var/lib/nova/instances/f73def3d-d74d-4560-8189-60f0b20e8a87# qemu-img check disk
qemu-img: Could not open 'disk': Could not open backing file: Could not open '/var/lib/nova/instances/_base/8d885c398b6b408be90dc5a88f6a4ae2d498f729': No such file or directory

The image very much appears to be gone. My only guess is that I was doing some housecleaning and deleted it...except I would have thought that I would have been prevented from deleting it if it was in use. So two questions:

  1. Is there any way to recover this VM?
  2. Is there really no safeguard against deleting an image that's in use, or would I have had to use something like --force?
edit retag flag offensive close merge delete

Comments

You should be able to delete an image from Glance's image store without affecting the instances that were created from it. However, in your case what's missing is the copy of the image on the compute node where your instance is located. Someone/something deleted this copy under /var/lib/nova.

Bernd Bausch gravatar imageBernd Bausch ( 2020-07-17 23:06:11 -0600 )edit

The _base directory contains backing files for qcow files used by instances. I fear that without this _base file, you won't be able to resurrect your instance. How did it disappear? I can't say, but in any case, OpenStack can't prevent you from simply removing files from the filesystem.

Bernd Bausch gravatar imageBernd Bausch ( 2020-07-17 23:10:13 -0600 )edit

See also https://docs.openstack.org/nova/train....

The error message seems to indicate that Nova notices the absence of the file under _base and tries to recreated it from a Glance image. Since that Glance image is gone as well, it gives up.

Bernd Bausch gravatar imageBernd Bausch ( 2020-07-17 23:12:52 -0600 )edit

There is some "garbage collector" process which seems to harvest base images on the compute nodes. Openstack Ansible installed it, so I'm not quite sure what the criteria are. The /var/lib/nova directory is shared, tho, so it could have been a different compute node deleting it.

bnordgren gravatar imagebnordgren ( 2020-07-18 08:51:08 -0600 )edit

According to the above documentation, "unused" base images are deleted. The term "unused" is not defined; a superficial look at the source indicates that "used" means "there is an instance", no matter in what state. But I might be wrong.

In any case, I fear your instance can't be restarted.

Bernd Bausch gravatar imageBernd Bausch ( 2020-07-18 20:11:06 -0600 )edit