Error on launching a LUKS volume on Queens: volume attachment <xxxxxx> could not be found
I need to implement encryption at rest on Queens. Followed the instructions on these pages to setup LUKS: https://docs.openstack.org/newton/con... https://docs.openstack.org/project-in...
I then create a volume using the GUI: Image Name: Centos7_1907 Use image as source: Centos7_1907 (this is a qcow2 image) Type: LUKS Size: 80GB
Volume is created successfully. I then select Launch as Instance using the volume above as the boot source. On the Instance page I see an error popup: "Centos7_1907", the instance has an error status: Please try again later [Error: Build of instance 07132fc4-233f-47cb-96f2-bffc751936a6 aborted: Volume attachment 9c38d98e-4171-414a-9091-37e65b6bd845 could not be found.
Where should I start looking for the cause of this error message? Thanks in advance.
Since comments are limited to 300 chars, i'm replying here. Full error message from nova-conductor.log:
nova-conductor.log:2019-09-27 11:08:23.794 31268 ERROR nova.scheduler.utils [req-eccb5cff-265d-4558-be46-79860d7061cc 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - default default] [instance: 12b6cc15-3810-40b6-8b7c-e82ea99c44a2] Error from last host: compute1 (node compute1): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1840, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2117, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 12b6cc15-3810-40b6-8b7c-e82ea99c44a2 was re-scheduled: No module named barbicanclient\n']
In cinder logs:
volume.log:2019-09-27 11:05:36.619 165666 WARNING stevedore.named [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager
volume.log:2019-09-27 11:05:36.620 165666 WARNING castellan.key_manager [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager'
volume.log:2019-09-27 11:05:43.814 165666 WARNING stevedore.named [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager
volume.log:2019-09-27 11:05:43.815 165666 WARNING castellan.key_manager [req-4c932006-76f3-4033-ba8f-4102da5b531a 5131e1277a664ef0b452a58192c17145 63f96a537d7e484a8ea7ebcc34dc6ba6 - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager'
I've also checked castellan:
[root@controller cinder]# pip install castellan
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: castellan in /usr/lib/python2.7/site-packages (0.17.0)
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/lib/python2.7/site-packages (from castellan) (3.1.1)
Requirement already satisfied: Babel!=2.4.0,>=2.3.4 in /usr/lib/python2.7/site-packages (from castellan) (2.3.4)
Requirement already satisfied: cryptography!=2.0,>=1.9 in /usr ...
You start looking in the log files. In this case, I would start with the Nova compute and Cinder API logs. To narrow down your search, focus on ERROR strings and the volume's and instance's UUIDs.
In nova-conductor.log I see ERROR nova.scheduler.utils ..... RescheduledException: Build of instance ... was re-scheduled: No module named barbicanclient.
I have barbican-client installed but there is no include for barbican-client in manager.py. Do I need to add it?
The required Python package is named python-barbicanclient. I guess it has to be on the compute node (not certain though). No, I can't imagine that you need to add it to manager.py.
Can you provide the full error message? Are there other Barbican errors in other Nova log files?
Question: after creating an encrypted volume should I see a key in the keystore? I get nothing when executing 'openstack secret list' I'm thinking something is not setup correctly with barbican.
I have no idea, but a search for the error No 'castellan.drivers' driver found comes back with a few promising-looking results.