Ask Your Question
0

how to debug kolla cinder lvm2 with iscsi

asked 2019-11-01 19:11:31 -0600

vpradhan gravatar image

Hi,

After a reboot one of the cinder backends is down (I have two backends). From looking at the docker logs I'm guessing cinder volume is not working as expected.

nova_compute logs has the following issues:

2019-11-01 18:54:16.513 6 INFO os_brick.initiator.connectors.iscsi [req-452a07f5-5e84-4719-a877-a00f74b20c2e 8725533bfac744dd9799af404e433795 64d7a9d9af7947869344aff6a66d8a0f - default default] Trying to connect to iSCSI portal 10.10.10.230:3260
2019-11-01 18:56:34.435 6 WARNING os_brick.initiator.connectors.iscsi [req-452a07f5-5e84-4719-a877-a00f74b20c2e 8725533bfac744dd9799af404e433795 64d7a9d9af7947869344aff6a66d8a0f - default default] Failed to login iSCSI target iqn.2010-10.org.openstack:volume-7bda0cb4-cbc8-48c4-a432-b1981135f6be on portal 10.10.10.230:3260 (exit code 11).: ProcessExecutionError: Unexpected error while running command.
2019-11-01 18:56:34.437 6 WARNING os_brick.initiator.connectors.iscsi [req-452a07f5-5e84-4719-a877-a00f74b20c2e 8725533bfac744dd9799af404e433795 64d7a9d9af7947869344aff6a66d8a0f - default default] Failed to connect to iSCSI portal 10.10.10.230:3260.
2019-11-01 19:01:25.242 6 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 584, in _connect_single_volume
2019-11-01 19:01:25.242 6 ERROR oslo_messaging.rpc.server     raise exception.VolumeDeviceNotFound(device='')
2019-11-01 19:01:25.242 6 ERROR oslo_messaging.rpc.server VolumeDeviceNotFound: Volume device not found at .

tgtd logs has this:

Running command: 'tgtd -d 1 -f --iscsi portal=10.10.10.230:3260'
+ echo 'Running command: '\''tgtd -d 1 -f --iscsi portal=10.10.10.230:3260'\'''
+ exec tgtd -d 1 -f --iscsi portal=10.10.10.230:3260
tgtd: iser_ib_init(3436) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_start(146) use timer_fd based scheduler
tgtd: work_timer_start(181) started, timeout: 500 msec
tgtd: bs_init_signalfd(267) could not open backing-store module directory /usr/lib64/tgt/backing-store
tgtd: bs_init(386) use signalfd notification

Please let me know if anyone has any ideas.

Thanks in advance.

edit retag flag offensive close merge delete

Comments

When you enable debug logging on the compute node, nova-compute will probably log the precise command that leads to the ProcessExecutionError. It may contain clues.

You should also try to find out what's wrong with /usr/lib64/tgt/backing-store on the iSCSI target.

Bernd Bausch gravatar imageBernd Bausch ( 2019-11-02 00:22:01 -0600 )edit

I found out that the tgtd container has no targets on the faulty cinder node.

(tgtd)[root@zeus /]# tgtadm --lld iscsi --op show --mode target
(tgtd)[root@zeus /]#

Does anyone know how to get these back?

vpradhan gravatar imagevpradhan ( 2019-11-05 16:58:38 -0600 )edit

tgtd complains about /usr/lib64/tgt/backing-store. This is where I would investigate. Off the cuff, I don't know the role of this directory.

How are the two backends defined?

Bernd Bausch gravatar imageBernd Bausch ( 2019-11-05 17:37:29 -0600 )edit

Thanks for the help. I spent a long time trying to figure out how to start debug mode for a kolla installation :( But found the solution on the way.

vpradhan gravatar imagevpradhan ( 2019-11-05 17:59:33 -0600 )edit

Fixed by trying the do what this link says https://ask.openstack.org/en/question...

First in the GUI go to admin and set the state of the volume to be in-use. Then docker restart cinder_volume tgtd iscsid

vpradhan gravatar imagevpradhan ( 2019-11-05 18:01:52 -0600 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2019-11-15 16:22:25 -0600

vpradhan gravatar image

Fixed by trying the do what this link says https://ask.openstack.org/en/question...

First in the GUI go to admin and set the state of the volume to be in-use. Then docker restart cinder_volume tgtd iscsid This restarts all cinder related docker container. Not sure if this is overkill.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-11-01 19:11:31 -0600

Seen: 35 times

Last updated: Nov 15