Cinder disaster recovery procedure?
Hi folks!
Recently, after attempting to recover from disaster, I ended up with an inconsistent cinder database due to incorrect manual editing. :-(
However, I devised a procedure to obtain a clean cinder db without losing data from the volumes. It seems to work, but I'd like some expert opinion to see if I'm not missing something.
Here are the steps:
- dump all volume data (cinder list, cinder show etc).
- stop all cinder services
- delete all iscsi targets (tgt-admin --delete)
- go into mysql and drop database cinder
- recreate cinder database and permissions, and do cinder-manage db sync
- restart all cinder services
- for each volume in the cinder LVM group:
- create new volume through cinder with same size and metadata
- stop all cinder services
- delete new volume's iscsi target
- lvremove new volume
- lvrename old volume to new volume's name
- restart all cinder services
- back to step 7/8 until all volumes are recovered
This seems to work like a charm in my experience, but I still worry it may be creating some inconsistencies in cinder's/tgt metadata that I'm unaware of.
Can any of you experts analyze this and tell me if this is a sound procedure?
Thanks!
-- thiago