Cinder disaster recovery procedure?

asked 2013-10-27 09:43:33 -0500

sigrist gravatar image

Hi folks!

Recently, after attempting to recover from disaster, I ended up with an inconsistent cinder database due to incorrect manual editing. :-(

However, I devised a procedure to obtain a clean cinder db without losing data from the volumes. It seems to work, but I'd like some expert opinion to see if I'm not missing something.

Here are the steps:

  1. dump all volume data (cinder list, cinder show etc).
  2. stop all cinder services
  3. delete all iscsi targets (tgt-admin --delete)
  4. go into mysql and drop database cinder
  5. recreate cinder database and permissions, and do cinder-manage db sync
  6. restart all cinder services
  7. for each volume in the cinder LVM group:
  8. create new volume through cinder with same size and metadata
  9. stop all cinder services
  10. delete new volume's iscsi target
  11. lvremove new volume
  12. lvrename old volume to new volume's name
  13. restart all cinder services
  14. back to step 7/8 until all volumes are recovered

This seems to work like a charm in my experience, but I still worry it may be creating some inconsistencies in cinder's/tgt metadata that I'm unaware of.

Can any of you experts analyze this and tell me if this is a sound procedure?

Thanks!

-- thiago

edit retag flag offensive close merge delete