Cinder volumes hanging/wedged

asked 2016-01-05 12:35:18 -0500

txsprout gravatar image

This is a juno installation. We are seeing VMs with cinder nodes attached mark the nova-compute service as down when running /sbin/blockdev. In this example pid 16698 is hung/wedged.

_ /usr/bin/python /usr/local/bin/nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /dev/disk/by-path/ip-xxx.xxx.xxx.xxx... root 16698 0.0 0.0 4336 360 ? D 2015 0:00 _ /sbin/blockdev --getsize64 /dev/disk/by-path/ip-xxx.xxx.xxx.xxx:3260-iscsi-iqn.2010-10.org.openstack:xxx...

iscsadm -m session shows the iSCSI session as active, but it clearly as not. Trying to strace the process, or access the iscsi volume results in hanging those processes as well.

There are no errors in cinder or dmesg or syslog to indicate any issues.

The storage network is 10GB, does not use jumbo frames, but we have not seen anything indicative of network congestion.

Has anyone else seen this?

edit retag flag offensive close merge delete