Invalid Shared Storage error on live-migration with GlusterFS
I have an Openstack cluster with 1 controller and 2 compute nodes at https://www.cloudlab.us/ (CloudLab) and I am trying to enable live-migration with GlusterFS shared storage following the steps listed http://yongluo2013.github.io/openstack-vm-live-migration/ (here).
I have setup the GlusterFS Volume and it is mounted at /var/lib/nova/instances
.
root@cp-2:/users/Jasim9# gluster volume info
Volume Name: nova-gluster-vol
Type: Replicate
Volume ID: ae2baad8-0363-4391-8f24-86593648f7f9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: cp-1:/brick1/sdb
Brick2: cp-2:/brick1/sdb
Options Reconfigured:
performance.readdir-ahead: on
as can be seen in df
on compute nodes:
root@cp-1:/users/Jasim9# df -hT
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 126G 0 126G 0% /dev
tmpfs tmpfs 26G 34M 26G 1% /run
/dev/sda1 ext3 16G 2.9G 12G 20% /
tmpfs tmpfs 126G 0 126G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 126G 0 126G 0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0 nfs 100G 512K 100G 1% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share nfs 97G 1.2G 88G 2% /share
tmpfs tmpfs 26G 0 26G 0% /run/user/0
cgmfs tmpfs 100K 0 100K 0% /run/cgmanager/fs
/dev/sda4 ext4 896G 73M 850G 1% /var/lib/nova
tmpfs tmpfs 26G 0 26G 0% /run/user/20001
/dev/sdb xfs 932G 865M 931G 1% /brick1
cp-1:/nova-gluster-vol fuse.glusterfs 932G 865M 931G 1% /var/lib/nova/instances
root@cp-2:/users/Jasim9# df -hT
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 126G 0 126G 0% /dev
tmpfs tmpfs 26G 34M 26G 1% /run
/dev/sda1 ext3 16G 2.9G 12G 20% /
tmpfs tmpfs 126G 0 126G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 126G 0 126G 0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0 nfs 100G 512K 100G 1% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share nfs 97G 1.2G 88G 2% /share
tmpfs tmpfs 26G 0 26G 0% /run/user/0
cgmfs tmpfs 100K 0 100K 0% /run/cgmanager/fs
/dev/sda4 ext4 896G 73M 850G 1% /var/lib/nova
tmpfs tmpfs 26G 0 26G 0% /run/user/20001
/dev/sdb xfs 932G 885M 931G 1% /brick1
cp-2:/nova-gluster-vol fuse.glusterfs 932G 885M 931G 1% /var/lib/nova/instances
I have added the necessary commands in /etc/libvirt/libvirtd.conf
on both compute nodes:
listen_tls = 0
listen_tcp = 1
auth_tcp = “none”
and added
# options passed to libvirtd, add "-l" to listen on tcp
libvirtd_opts="-l"
to /etc/default/libvirt-bin
and the flags in /etc/nova/nova.conf
on both compute nodes:
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
Have created two VM instances (test2 on cp-1 test on cp-2):
root@ctl:/users/Jasim9# nova list
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
| a9e80a50-f849-44e9-9e04-2a5a89ed7f43 | test | ACTIVE | - | Running | flat-lan-1-net=10.11.10.10 |
| b0de71ab-75df-4c21-9648-e69c47bd43e2 | test2 | ACTIVE | - | Running | flat-lan-1-net=10.11.10.11 |
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
but when trying to live-migrate them to other host ...