HA: Recovering from Cinder node loss with Ceph backend

asked 2014-12-04 13:50:42 -0600

dmmatson gravatar image

updated 2014-12-04 15:08:25 -0600

Given I am running Openstack (Icehouse 2014.1.3) in active/active HA with Ceph as the storage backend, and I have booted a VM from a cloned snapshot, is it expected behavior that I cannot snapshot or delete the clone if I lose the controller that initially created the volume?

If so, is the only recovery method to manually change the host field in the database to an active controller?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2014-12-04 16:33:33 -0600

SamYaple gravatar image

updated 2014-12-05 15:43:13 -0600

Currently, the cinder-volume service is very much _not_ HA. If you lose the cinder-volume service that created the volume you will run into a host of issues including the one you describe. You can only run it in active-passive.

Here is how to properly configure Cinder-Volume in a multinode environment with ceph as of Juno:


  • 2 nodes (can be more) both running Cinder-Volume service backed by a distributed backend like Ceph
  • Both nodes have a generic hostname that is the same (e.g. "controller" not "controller01")


  1. Install cinder-volume services on both controllers.
  2. Shut down cinder-volume on one of the controllers (must be shutdown completely, not running)
  3. Check cinder service-list and ensure you only see one cinder-volume binary running with our generic hostname
  4. Configure Corosync/Pacemaker or similar to start a service on the inactive controller when the active controller fails

Then you should have an HA solution. The hostname part is what is going to trip you up. It has to be the same name so cinder sees it as the same service.

I am told this is all being worked on for Kilo. If it isn't fixed in Kilo, it should be by the time we get to "L"

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-12-04 13:50:42 -0600

Seen: 1,068 times

Last updated: Dec 05 '14