Ask Your Question

Cinder causes LVM to lock

asked 2014-06-03 07:52:22 -0500

aweaver614 gravatar image

updated 2014-06-03 08:14:41 -0500

SamYaple gravatar image


I have Icehouse deployed on Ubuntu 12.04.4 with a dedicated Cinder node using LVs.

A user reported that deleting/creating volumes was 'stuck' in Horizon overnight and this is what I am seeing on the cinder node:

5122 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g
 5123 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g
 5124 ?        S<L    0:00 /sbin/lvcreate -n volume-cec41e3b-060b-4066-8563-6e6f4e39b554 cinder -L 5g
 5129 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
 5130 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
 5131 ?        S      0:00 /sbin/vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
 5135 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g
 5136 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g
 5137 ?        S      0:00 /sbin/lvcreate -n volume-22400284-aec6-493e-9c7b-21253cefe050 cinder -L 5g
 5152 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8
 5155 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8
 5160 ?        S      0:00 /sbin/lvchange -y -an cinder/volume-06e7b9c0-82cf-4e64-a392-805efb7a71b8
 7167 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g
 7168 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g
 7169 ?        S      0:00 /sbin/lvcreate -n volume-9db456b0-4fae-4e20-b8f4-9a41704ff25a cinder -L 2g
 7987 pts/2    R+     0:00 grep --color=auto cinder
11989 ?        Ss    42:18 /usr/bin/python /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log
12000 ?        S     17:37 /usr/bin/python /usr/bin/cinder-volume --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-volume.log
18192 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g
18193 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g
18194 ?        S      0:00 /sbin/lvcreate -n volume-d6dc8cc8-1e04-4e11-9fc1-f1e75767510d cinder -L 10g
19794 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g
19795 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g
19796 ?        S      0:00 /sbin/lvcreate -n volume-817034dc-e9d6-475e-be03-b0b830d3d15d cinder -L 10g
32143 ?        S      0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g
32144 ?        S      0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g
32145 ?        S      0:00 /sbin/lvcreate -n volume-107c6662-9966-4916-a656-31b738e03d43 cinder -L 5g

I also noticed that any non-pv related ... (more)

edit retag flag offensive close merge delete


What are your cinder logs saying? You appear to have just given ps output. Likely your logs may reveal what is hanging. You may also need to turn on verbose and possible debug in your cinder.conf

That being said, my initial place to look would be the actual disks. What is accessing them, have any dropped offline, etc. Assuming that is all good, my money would be on a udev issue with LVM. Restarting the appropriate services (the whole node would be better if it is isolated) would correct the issue. You would need to investigate further after that.

SamYaple gravatar imageSamYaple ( 2014-06-03 08:21:23 -0500 )edit

2 answers

Sort by ยป oldest newest most voted

answered 2014-06-03 08:57:26 -0500

Duncan Thomas gravatar image

Certainly a copy of cinder-volume.log would be extremely useful here

edit flag offensive delete link more

answered 2014-09-24 07:43:47 -0500

semyazz gravatar image

Same problem. I wanted to create few volumes at the same time and ended up in the same state. LVS hangs because it can't obtain a lock, because cinder has it. Any idea how to fix that?

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-06-03 07:52:22 -0500

Seen: 430 times

Last updated: Sep 24 '14