After shutting down / powering up Ceph cluster, cannot start instances with encrypted volumes

asked 2019-03-22 10:08:49 -0500

maglo gravatar image

I have enabled support for encrypted volumes in OpenStack Rocky. I use Ceph as storage backend.

There was no problem creating an encrypted volume and attaching it to an instance and filling it with data.

Later, there was a need for powering down the entire cluster for renovation work in the server room. The Ceph cluster was taken down according to these instructions:

https://ceph.com/planet/how-to-do-a-ceph-cluster-maintenance-shutdown/ (https://ceph.com/planet/how-to-do-a-c...)

Instances without encrypted volumes start nicely after the power up procedure.

However, instances with encrypted volumes do not start and this can be seen in the log files:

https://pastebin.com/mvMbJQRb

Any suggestions?

As you can imagine, these servers with encrypted volumes are the important ones...

edit retag flag offensive close merge delete

Comments

Is the ceph cluster in a healthy state after the maintenance? Maybe the ceph logs reveal something helpful. Are the volumes accessible in the ceph cluster? Has anything changed in you configs, maybe something like rbd secret or anything else?

eblock gravatar imageeblock ( 2019-03-25 04:04:41 -0500 )edit

Is the ceph cluster in a healthy state after the maintenance

Yes, instances with unencrypted volumes start fine

Maybe the ceph logs reveal something helpful. I will check, I doubt it though...

maglo gravatar imagemaglo ( 2019-03-25 08:29:10 -0500 )edit

Are the volumes accessible in the ceph cluster?

I believe so, the instances with unencryped volumes start fine...

Has anything changed in you configs, maybe something like rbd secret or anything else No...

maglo gravatar imagemaglo ( 2019-03-25 08:30:05 -0500 )edit

Filed a bug https://bugs.launchpad.net/nova/+bug/1821696 (https://bugs.launchpad.net/nova/+bug/...)

maglo gravatar imagemaglo ( 2019-03-26 03:18:58 -0500 )edit