Error on launching a LUKS volume on Queens: volume attachment <xxxxxx> could not be found

asked 2019-09-25 09:21:18 -0600

updated 2019-09-30 19:11:17 -0600

I need to implement encryption at rest on Queens. Followed the instructions on these pages to setup LUKS: https://docs.openstack.org/newton/con... https://docs.openstack.org/project-in...

I then create a volume using the GUI: Image Name: Centos7_1907 Use image as source: Centos7_1907 (this is a qcow2 image) Type: LUKS Size: 80GB

Volume is created successfully. I then select Launch as Instance using the volume above as the boot source. On the Instance page I see an error popup: "Centos7_1907", the instance has an error status: Please try again later [Error: Build of instance 07132fc4-233f-47cb-96f2-bffc751936a6 aborted: Volume attachment 9c38d98e-4171-414a-9091-37e65b6bd845 could not be found.

Where should I start looking for the cause of this error message? Thanks in advance.


Since comments are limited to 300 chars, i'm replying here. Full error message from nova-conductor.log:

nova-conductor.log:2019-09-27 11:08:23.794 31268 ERROR nova.scheduler.utils [req-eccb5cff-265d-4558-be46-79860d7061cc 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - default default] [instance: 12b6cc15-3810-40b6-8b7c-e82ea99c44a2] Error from last host: compute1 (node compute1): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1840, in _do_build_and_run_instance\n    filter_properties, request_spec)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2117, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 12b6cc15-3810-40b6-8b7c-e82ea99c44a2 was re-scheduled: No module named barbicanclient\n']

In cinder logs:

volume.log:2019-09-27 11:05:36.619 165666 WARNING stevedore.named [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager
volume.log:2019-09-27 11:05:36.620 165666 WARNING castellan.key_manager [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager'
volume.log:2019-09-27 11:05:43.814 165666 WARNING stevedore.named [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager
volume.log:2019-09-27 11:05:43.815 165666 WARNING castellan.key_manager [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager'

I've also checked castellan:

[root@controller cinder]# pip install castellan
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: castellan in /usr/lib/python2.7/site-packages (0.17.0)
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/lib/python2.7/site-packages (from castellan) (3.1.1)
Requirement already satisfied: Babel!=2.4.0,>=2.3.4 in /usr/lib/python2.7/site-packages (from castellan) (2.3.4)
Requirement already satisfied: cryptography!=2.0,>=1.9 in /usr ...
(more)
edit retag flag offensive close merge delete

Comments

You start looking in the log files. In this case, I would start with the Nova compute and Cinder API logs. To narrow down your search, focus on ERROR strings and the volume's and instance's UUIDs.

Bernd Bausch gravatar imageBernd Bausch ( 2019-09-25 19:09:37 -0600 )edit

In nova-conductor.log I see ERROR nova.scheduler.utils ..... RescheduledException: Build of instance ... was re-scheduled: No module named barbicanclient.

I have barbican-client installed but there is no include for barbican-client in manager.py. Do I need to add it?

mjpsammon gravatar imagemjpsammon ( 2019-09-26 08:26:40 -0600 )edit

The required Python package is named python-barbicanclient. I guess it has to be on the compute node (not certain though). No, I can't imagine that you need to add it to manager.py.

Can you provide the full error message? Are there other Barbican errors in other Nova log files?

Bernd Bausch gravatar imageBernd Bausch ( 2019-09-26 16:40:14 -0600 )edit

Question: after creating an encrypted volume should I see a key in the keystore? I get nothing when executing 'openstack secret list' I'm thinking something is not setup correctly with barbican.

mjpsammon gravatar imagemjpsammon ( 2019-09-27 12:43:41 -0600 )edit

I have no idea, but a search for the error No 'castellan.drivers' driver found comes back with a few promising-looking results.

Bernd Bausch gravatar imageBernd Bausch ( 2019-09-30 19:15:46 -0600 )edit