Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Cinder causes LVM to lock

Hi,

I have Icehouse deployed on Ubuntu 12.04.4 with a dedicated Cinder node using LVs.

A user reported that deleting/creating volumes was 'stuck' in Horizon overnight and this is what I am seeing on the cinder node:

5122 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g 5123 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g 5124 ? S<l 0:00="" sbin="" lvcreate="" -n="" volume-cec41e3b-060b-4066-8563-6e6f4e39b554="" cinder="" -l="" 5g="" 5129="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" env="" lc_all="C" vgs="" --noheadings="" --unit="g" -o="" name,size,free,lv_count,uuid="" --separator="" :="" --nosuffix="" cinder="" 5130="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" env="" lc_all="C" vgs="" --noheadings="" --unit="g" -o="" name,size,free,lv_count,uuid="" --separator="" :="" --nosuffix="" cinder="" 5131="" ?="" s="" 0:00="" sbin="" vgs="" --noheadings="" --unit="g" -o="" name,size,free,lv_count,uuid="" --separator="" :="" --nosuffix="" cinder="" 5135="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-22400284-aec6-493e-9c7b-21253cefe050="" cinder="" -l="" 5g="" 5136="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-22400284-aec6-493e-9c7b-21253cefe050="" cinder="" -l="" 5g="" 5137="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-22400284-aec6-493e-9c7b-21253cefe050="" cinder="" -l="" 5g="" 5152="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvchange="" -y="" -an="" cinder="" volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8="" 5155="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvchange="" -y="" -an="" cinder="" volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8="" 5160="" ?="" s="" 0:00="" sbin="" lvchange="" -y="" -an="" cinder="" volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8="" 7167="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a="" cinder="" -l="" 2g="" 7168="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a="" cinder="" -l="" 2g="" 7169="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a="" cinder="" -l="" 2g="" 7987="" pts="" 2="" r+="" 0:00="" grep="" --color="auto" cinder="" 11989="" ?="" ss="" 42:18="" usr="" bin="" python="" usr="" bin="" cinder-volume="" --config-file="/etc/cinder/cinder.conf" --log-file="/var/log/cinder/cinder-volume.log" 12000="" ?="" s="" 17:37="" usr="" bin="" python="" usr="" bin="" cinder-volume="" --config-file="/etc/cinder/cinder.conf" --log-file="/var/log/cinder/cinder-volume.log" 18192="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d="" cinder="" -l="" 10g="" 18193="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d="" cinder="" -l="" 10g="" 18194="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d="" cinder="" -l="" 10g="" 19794="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-817034dc-e9d6-475e-be03-b0b830d3d15d="" cinder="" -l="" 10g="" 19795="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-817034dc-e9d6-475e-be03-b0b830d3d15d="" cinder="" -l="" 10g="" 19796="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-817034dc-e9d6-475e-be03-b0b830d3d15d="" cinder="" -l="" 10g="" 32143="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-107c6662-9966-4916-a656-31b738e03d43="" cinder="" -l="" 5g="" 32144="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-107c6662-9966-4916-a656-31b738e03d43="" cinder="" -l="" 5g="" 32145="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-107c6662-9966-4916-a656-31b738e03d43="" cinder="" -l="" 5g<="" p="">

I also noticed that any non-pv related LVM commands hang until they are killed:

root@node-3:/var/log# lvscan ^C CTRL-c detected: giving up waiting for lock /var/lock/lvm/V_cinder:aux: flock failed: Interrupted system call Can't get lock for cinder Skipping volume group cinder

Cinder causes LVM to lock

Hi,

Hi,
  

I have Icehouse deployed on Ubuntu 12.04.4 with a dedicated Cinder node using LVs.

LVs.

A user reported that deleting/creating volumes was 'stuck' in Horizon overnight and this is what I am seeing on the cinder node:

node:

5122 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g 5123 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g 5124 ? S<l 0:00="" sbin="" lvcreate="" -n="" volume-cec41e3b-060b-4066-8563-6e6f4e39b554="" cinder="" -l="" 5g="" 5129="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" env="" lc_all="C" vgs="" --noheadings="" --unit="g" -o="" name,size,free,lv_count,uuid="" --separator="" :="" --nosuffix="" cinder="" 5130="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" env="" lc_all="C" vgs="" --noheadings="" --unit="g" -o="" name,size,free,lv_count,uuid="" --separator="" :="" --nosuffix="" cinder="" 5131="" ?="" s="" 0:00="" sbin="" vgs="" --noheadings="" --unit="g" -o="" name,size,free,lv_count,uuid="" --separator="" :="" --nosuffix="" cinder="" 5135="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-22400284-aec6-493e-9c7b-21253cefe050="" cinder="" -l="" 5g="" 5136="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-22400284-aec6-493e-9c7b-21253cefe050="" cinder="" -l="" 5g="" 5137="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-22400284-aec6-493e-9c7b-21253cefe050="" cinder="" -l="" 5g="" 5152="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvchange="" -y="" -an="" cinder="" volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8="" 5155="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvchange="" -y="" -an="" cinder="" volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8="" 5160="" ?="" s="" 0:00="" sbin="" lvchange="" -y="" -an="" cinder="" volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8="" 7167="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a="" cinder="" -l="" 2g="" 7168="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a="" cinder="" -l="" 2g="" 7169="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a="" cinder="" -l="" 2g="" 7987="" pts="" 2="" r+="" 0:00="" grep="" --color="auto" cinder="" 11989="" ?="" ss="" 42:18="" usr="" bin="" python="" usr="" bin="" cinder-volume="" --config-file="/etc/cinder/cinder.conf" --log-file="/var/log/cinder/cinder-volume.log" 12000="" ?="" s="" 17:37="" usr="" bin="" python="" usr="" bin="" cinder-volume="" --config-file="/etc/cinder/cinder.conf" --log-file="/var/log/cinder/cinder-volume.log" 18192="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d="" cinder="" -l="" 10g="" 18193="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d="" cinder="" -l="" 10g="" 18194="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d="" cinder="" -l="" 10g="" 19794="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-817034dc-e9d6-475e-be03-b0b830d3d15d="" cinder="" -l="" 10g="" 19795="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-817034dc-e9d6-475e-be03-b0b830d3d15d="" cinder="" -l="" 10g="" 19796="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-817034dc-e9d6-475e-be03-b0b830d3d15d="" cinder="" -l="" 10g="" 32143="" ?="" s="" 0:00="" sudo="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-107c6662-9966-4916-a656-31b738e03d43="" cinder="" -l="" 5g="" 32144="" ?="" s="" 0:00="" usr="" bin="" python="" usr="" bin="" cinder-rootwrap="" etc="" cinder="" rootwrap.conf="" lvcreate="" -n="" volume-107c6662-9966-4916-a656-31b738e03d43="" cinder="" -l="" 5g="" 32145="" ?="" s="" 0:00="" sbin="" lvcreate="" -n="" volume-107c6662-9966-4916-a656-31b738e03d43="" cinder="" -l="" 5g<="" p=""> S<L 0:00 /sbin/lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g 5129 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder 5130 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder 5131 ? S 0:00 /sbin/vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder 5135 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g 5136 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g 5137 ? S 0:00 /sbin/lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g 5152 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8 5155 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8 5160 ? S 0:00 /sbin/lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8 7167 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g 7168 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g 7169 ? S 0:00 /sbin/lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g 7987 pts/2 R+ 0:00 grep --color=auto cinder 11989 ? Ss 42:18 /usr/bin/python /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log 12000 ? S 17:37 /usr/bin/python /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log 18192 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g 18193 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g 18194 ? S 0:00 /sbin/lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g 19794 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g 19795 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g 19796 ? S 0:00 /sbin/lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g 32143 ? S 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g 32144 ? S 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g 32145 ? S 0:00 /sbin/lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g

I also noticed that any non-pv related LVM commands hang until they are killed:

killed:

root@node-3:/var/log# lvscan ^C CTRL-c detected: giving up waiting for lock /var/lock/lvm/V_cinder:aux: flock failed: Interrupted system call Can't get lock for cinder Skipping volume group cinder

cinder
click to hide/show revision 3
No.3 Revision

Cinder causes LVM to lock

Hi,

Hi,

I have Icehouse deployed on Ubuntu 12.04.4 with a dedicated Cinder node using LVs. LVs.

A user reported that deleting/creating volumes was 'stuck' in Horizon overnight and this is what I am seeing on the cinder node: node:

5122 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g
 5123 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g
 5124 ?        S<L    0:00 /sbin/lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g
 5129 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
 5130 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
 5131 ?        S      0:00 /sbin/vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
 5135 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g
 5136 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g
 5137 ?        S      0:00 /sbin/lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g
 5152 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8
 5155 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8
 5160 ?        S      0:00 /sbin/lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8
 7167 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g
 7168 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g
 7169 ?        S      0:00 /sbin/lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g
 7987 pts/2    R+     0:00 grep --color=auto cinder
11989 ?        Ss    42:18 /usr/bin/python /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log
12000 ?        S     17:37 /usr/bin/python /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log
18192 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g
18193 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g
18194 ?        S      0:00 /sbin/lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g
19794 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g
19795 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g
19796 ?        S      0:00 /sbin/lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g
32143 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g
32144 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g
32145 ?        S      0:00 /sbin/lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g
 

I also noticed that any non-pv related LVM commands hang until they are killed: killed:

root@node-3:/var/log# lvscan
^C  CTRL-c detected: giving up waiting for lock
  /var/lock/lvm/V_cinder:aux: flock failed: Interrupted system call
  Can't get lock for cinder
  Skipping volume group cinder