nova rebuild fails when the instance has ceph snapshot
Hi Team,
We have openstack with ceph storage backend. We use ceph snapshot as our backup strategy for backing up root and additional drives which are attached. But what we have observed is whenever we do nova rebuild on a server which is having a snapshot created in ceph, the rebuld completes without error but actual rebuild doesn't happen at all. Post rebuild we still have the same corrupted VM or at times we rebuild the vm to a fresh new OS like from Ubuntu to centos, but after rebuild we still would have same Ubuntu. In all the cases what we have in common is ceph is having a snapshot for the instance root drives. For instances which are not having ceph snap, the rebuild just works like a charm.
Has anyone faced this kind of issue. Please guide how to resolve this.
Regards, Ram.
I haven't done much rebuilding, so my experience is limited here. But have you turned on debug logs for nova? I'd expect to see the commands it's trying to execute, maybe there's a hint what could have gone wrong. If I have the time I'll try to reproduce that.
Your description is accurate, I was able to reproduce this. I'll try to find out more.