cinder kolla-volume attach error-openstack stein

asked 2020-05-11 09:12:08 -0500

sanjay khuntia gravatar image

Hi everyone,

I have deployed all openstack services charts (openstack stein) in my k8s cluster through openstack helm charts. Everything is running fine. Now when I try to attach a cinder volume to one centos/ubuntu instance through horizon cinder. However I get the below errors in kolla/cinder/cinder-volume.log.

ERROR oslo_messaging.rpc.server [req-da0f46d8-7c1c-485a-a8f5-830181bf6082 5548b68d163bde7ff0b691fa6ba9614d6c752a8bd2c2e1d548d69611b16b833c 6ae50fba42cb492799434ed670e1656d - 6b2dd4f8909845f789e62127cae28090 6b2dd4f8909845f789e62127cae28090] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Command: tgtadm --lld iscsi --op show --mode target
Exit code: 107
Stdout: u''
Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n'
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 4521, in attachment_update
ERROR oslo_messaging.rpc.server     connector)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 4449, in _connection_create
ERROR oslo_messaging.rpc.server     volume, connector)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 830, in create_export
ERROR oslo_messaging.rpc.server     volume_path)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
ERROR oslo_messaging.rpc.server     **portals_config)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/utils.py", line 818, in _wrapper
ERROR oslo_messaging.rpc.server     return r.call(f, *args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/retrying.py", line 206, in call
ERROR oslo_messaging.rpc.server     return attempt.get(self._wrap_exception)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/retrying.py", line 247, in get
ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/retrying.py", line 200, in call
ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 126, in create_iscsi_target
ERROR oslo_messaging.rpc.server     (out, err) = cinder.privsep.targets.tgt.tgtadm_show()
ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
ERROR oslo_messaging.rpc.server     return ...
(more)
edit retag flag offensive close merge delete

Comments

As a first troubleshooting step, I would run that command on the system and container where the volume is located: tgtadm --lld iscsi --op show --mode target. Perhaps tgtadm has a verbose flag that provides more info.

Bernd Bausch gravatar imageBernd Bausch ( 2020-05-11 10:12:50 -0500 )edit

I started the tgtd process from root user inside cinder-volume container. Now the above error is gone. But the volume is attached to instance for few seconds and then it is detached.

sanjay khuntia gravatar imagesanjay khuntia ( 2020-05-26 10:29:00 -0500 )edit

And there is nothing in the Cinder and Nova logs?

Bernd Bausch gravatar imageBernd Bausch ( 2020-05-26 17:43:54 -0500 )edit

The cinder logs show the volume attached and then detached after few seconds. i got the info from openstack kolla-ansible dev forum that the tgtd service should run in a separate container which I missed to deploy. answer

sanjay khuntia gravatar imagesanjay khuntia ( 2020-05-27 00:12:22 -0500 )edit