Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Invalid Shared Storage error on live-migration with GlusterFS

I have an Openstack cluster with 1 controller and 2 compute nodes at CloudLab and I am trying to enable live-migration with GlusterFS shared storage following the steps listed here. I have setup the GlusterFS Volume and it is mounted at /var/lib/nova/instances.

root@cp-2:/users/Jasim9# gluster volume info

Volume Name: nova-gluster-vol
Type: Replicate
Volume ID: ae2baad8-0363-4391-8f24-86593648f7f9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: cp-1:/brick1/sdb
Brick2: cp-2:/brick1/sdb
Options Reconfigured:
performance.readdir-ahead: on

as can be seen in df on compute nodes:

root@cp-1:/users/Jasim9# df -hT
Filesystem                                     Type            Size  Used Avail Use% Mounted on
udev                                           devtmpfs        126G     0  126G   0% /dev
tmpfs                                          tmpfs            26G   34M   26G   1% /run
/dev/sda1                                      ext3             16G  2.9G   12G  20% /
tmpfs                                          tmpfs           126G     0  126G   0% /dev/shm
tmpfs                                          tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs                                          tmpfs           126G     0  126G   0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0 nfs             100G  512K  100G   1% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share                 nfs              97G  1.2G   88G   2% /share
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/0
cgmfs                                          tmpfs           100K     0  100K   0% /run/cgmanager/fs
/dev/sda4                                      ext4            896G   73M  850G   1% /var/lib/nova
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/20001
/dev/sdb                                       xfs             932G  865M  931G   1% /brick1
cp-1:/nova-gluster-vol                         fuse.glusterfs  932G  865M  931G   1% /var/lib/nova/instances

root@cp-2:/users/Jasim9# df -hT
Filesystem                                     Type            Size  Used Avail Use% Mounted on
udev                                           devtmpfs        126G     0  126G   0% /dev
tmpfs                                          tmpfs            26G   34M   26G   1% /run
/dev/sda1                                      ext3             16G  2.9G   12G  20% /
tmpfs                                          tmpfs           126G     0  126G   0% /dev/shm
tmpfs                                          tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs                                          tmpfs           126G     0  126G   0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0 nfs             100G  512K  100G   1% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share                 nfs              97G  1.2G   88G   2% /share
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/0
cgmfs                                          tmpfs           100K     0  100K   0% /run/cgmanager/fs
/dev/sda4                                      ext4            896G   73M  850G   1% /var/lib/nova
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/20001
/dev/sdb                                       xfs             932G  885M  931G   1% /brick1
cp-2:/nova-gluster-vol                         fuse.glusterfs  932G  885M  931G   1% /var/lib/nova/instances

I have added the necessary commands in /etc/libvirt/libvirtd.conf on both compute nodes:

listen_tls = 0
listen_tcp = 1
auth_tcp = “none”

and added

# options passed to libvirtd, add "-l" to listen on tcp
libvirtd_opts="-l"

to /etc/default/libvirt-bin and the flags in /etc/nova/nova.conf on both compute nodes:

live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE

Have created two VM instances (test2 on cp-1 test on cp-2):

root@ctl:/users/Jasim9# nova list
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                   |
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
| a9e80a50-f849-44e9-9e04-2a5a89ed7f43 | test  | ACTIVE | -          | Running     | flat-lan-1-net=10.11.10.10 |
| b0de71ab-75df-4c21-9648-e69c47bd43e2 | test2 | ACTIVE | -          | Running     | flat-lan-1-net=10.11.10.11 |
+--------------------------------------+-------+--------+------------+-------------+----------------------------+

but when trying to live-migrate them to other host, e.g. for test:

root@ctl:/users/Jasim9# nova live-migration test cp-1.lm-stack.live-migrate-pg0.clemson.cloudlab.us                                               
ERROR (BadRequest): cp-2.lm-stack.live-migrate-pg0.clemson.cloudlab.us is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk. (HTTP 400) (Request-ID: req-81b3b929-a435-4ea0-b7d5-ff3d829698e6)

and in nova-conductor.log:

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
    payload)

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
    kwargs['instance'], e, sys.exc_info())

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5212, in check_can_live_migrate_source
    block_device_info)

  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5595, in check_can_live_migrate_source
    raise exception.InvalidSharedStorage(reason=reason, path=source)

InvalidSharedStorage: cp-2.lm-stack.live-migrate-pg0.clemson.cloudlab.us is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk.

I also tried the matching uid and gid configuration from this link, to match uid and gid of nova for controller and computes but it still does not solve the issue. I am not sure what have I missed, I did this about 2 weeks ago and it had worked then. I have tried restarting the libvirt-bin and nova-compute services on compute nodes and all nova services on controller but the issue persists.

Invalid Shared Storage error on live-migration with GlusterFS

I have an Openstack cluster with 1 controller and 2 compute nodes at CloudLab and I am trying to enable live-migration with GlusterFS shared storage following the steps listed here. I have setup the GlusterFS Volume and it is mounted at /var/lib/nova/instances.

root@cp-2:/users/Jasim9# gluster volume info

Volume Name: nova-gluster-vol
Type: Replicate
Volume ID: ae2baad8-0363-4391-8f24-86593648f7f9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: cp-1:/brick1/sdb
Brick2: cp-2:/brick1/sdb
Options Reconfigured:
performance.readdir-ahead: on

as can be seen in df on compute nodes:

root@cp-1:/users/Jasim9# df -hT
Filesystem                                     Type            Size  Used Avail Use% Mounted on
udev                                           devtmpfs        126G     0  126G   0% /dev
tmpfs                                          tmpfs            26G   34M   26G   1% /run
/dev/sda1                                      ext3             16G  2.9G   12G  20% /
tmpfs                                          tmpfs           126G     0  126G   0% /dev/shm
tmpfs                                          tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs                                          tmpfs           126G     0  126G   0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0 nfs             100G  512K  100G   1% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share                 nfs              97G  1.2G   88G   2% /share
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/0
cgmfs                                          tmpfs           100K     0  100K   0% /run/cgmanager/fs
/dev/sda4                                      ext4            896G   73M  850G   1% /var/lib/nova
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/20001
/dev/sdb                                       xfs             932G  865M  931G   1% /brick1
cp-1:/nova-gluster-vol                         fuse.glusterfs  932G  865M  931G   1% /var/lib/nova/instances

root@cp-2:/users/Jasim9# df -hT
Filesystem                                     Type            Size  Used Avail Use% Mounted on
udev                                           devtmpfs        126G     0  126G   0% /dev
tmpfs                                          tmpfs            26G   34M   26G   1% /run
/dev/sda1                                      ext3             16G  2.9G   12G  20% /
tmpfs                                          tmpfs           126G     0  126G   0% /dev/shm
tmpfs                                          tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs                                          tmpfs           126G     0  126G   0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0 nfs             100G  512K  100G   1% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share                 nfs              97G  1.2G   88G   2% /share
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/0
cgmfs                                          tmpfs           100K     0  100K   0% /run/cgmanager/fs
/dev/sda4                                      ext4            896G   73M  850G   1% /var/lib/nova
tmpfs                                          tmpfs            26G     0   26G   0% /run/user/20001
/dev/sdb                                       xfs             932G  885M  931G   1% /brick1
cp-2:/nova-gluster-vol                         fuse.glusterfs  932G  885M  931G   1% /var/lib/nova/instances

I have added the necessary commands in /etc/libvirt/libvirtd.conf on both compute nodes:

listen_tls = 0
listen_tcp = 1
auth_tcp = “none”

and added

# options passed to libvirtd, add "-l" to listen on tcp
libvirtd_opts="-l"

to /etc/default/libvirt-bin and the flags in /etc/nova/nova.conf on both compute nodes:

live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE

Have created two VM instances (test2 on cp-1 test on cp-2):

root@ctl:/users/Jasim9# nova list
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                   |
+--------------------------------------+-------+--------+------------+-------------+----------------------------+
| a9e80a50-f849-44e9-9e04-2a5a89ed7f43 | test  | ACTIVE | -          | Running     | flat-lan-1-net=10.11.10.10 |
| b0de71ab-75df-4c21-9648-e69c47bd43e2 | test2 | ACTIVE | -          | Running     | flat-lan-1-net=10.11.10.11 |
+--------------------------------------+-------+--------+------------+-------------+----------------------------+

but when trying to live-migrate them to other host, e.g. for test:

root@ctl:/users/Jasim9# nova live-migration test cp-1.lm-stack.live-migrate-pg0.clemson.cloudlab.us                                               
ERROR (BadRequest): cp-2.lm-stack.live-migrate-pg0.clemson.cloudlab.us is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk. (HTTP 400) (Request-ID: req-81b3b929-a435-4ea0-b7d5-ff3d829698e6)

and in nova-conductor.log:

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
    payload)

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
    kwargs['instance'], e, sys.exc_info())

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5212, in check_can_live_migrate_source
    block_device_info)

  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5595, in check_can_live_migrate_source
    raise exception.InvalidSharedStorage(reason=reason, path=source)

InvalidSharedStorage: cp-2.lm-stack.live-migrate-pg0.clemson.cloudlab.us is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk.

I also tried the matching uid and gid configuration from this link, to match uid and gid of nova for controller and computes but it still does not solve the issue. I am not sure what have I missed, I did this about 2 weeks ago and it had worked then. I have tried restarting the libvirt-bin and nova-compute services on compute nodes and all nova services on controller but the issue persists.