/var/lib/nova/instances on GlusterFS

asked 2014-05-20 02:41:08 -0500

OpenStackGossin gravatar image

I have a simple two node (storage01 and storage02) distributed Gluster (Gluster version 3.4) volume (no replica yet for capacity reasons but will introduce that this summer with additional storage nodes) and all my compute nodes mount the gluster volume on /var/lib/nova/instance. I run OpenStack Havana with four compute nodes.

(The setup is similar to the setup (section 5.3 and 5.4) in https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html-single/Configuring_Red_Hat_OpenStack_with_Red_Hat_Storage/index.html#idm73313664 (https://access.redhat.com/site/docume...) )

I run about 100 instances of Ubuntu server qcow image so about half of of the instances (since Gluster works this way) end up located on storage01 and the other half on storage02.

THE PROBLEM: All the instances are created from the same qcow base file which is located on storage01!

data@storage01:~$ ls -lh /brick0/_base/cda85378a9671ee52a774fe719daa868e78682d5
-rw-r--r-- 2 108 114 2.0G May 20 08:55 /brick0/_base/cda85378a9671ee52a774fe719daa868e78682d5

which means that all the instances on the other storage node (storage02) also use this file as their base file (which causes a lot of internal I/O traffic between storage nodes delaying I/O requests):

data@storage02:~$ file /brick0/34b6d389-1882-4038-b751-7d730653802f/disk 
/brick0/282a8552-accc-483b-aa5a-73cfb1229266/disk:       QEMU QCOW Image (v2), has backing file (path /var/lib/nova/instances/_base/cda85378a9671ee52a774fe719daa868e), 3221225472 bytes

I guess this is probably a case of performance vs capacity, since my instances consumes very little space, but I/O performance is bad.

I want to be able to do live migrations and do snapshots so I need my instances on a shared storage with a file format that supports snapshots, but I have to improve I/O performance.

Can I configure OpenStack to use qcow images but without a shared base file? (each instance will launch slower of course, but thats ok)

edit retag flag offensive close merge delete


I would avoid using gluster for the backend for the images for my vm's and go with NFS for my shared storage (for ephemeral vms) and Ceph for backend storage long-term vm's. I am currently using GlusterFS for Glance which works out pretty well

foster gravatar imagefoster ( 2015-05-16 06:42:02 -0500 )edit

thanks for the info foster, I ended up using only raw images for all my VMs, and it has been working ok, but Im switching all my storage to Ceph this summer since it's clear to me now that Ceph has become the defacto storage backend for OpenStack. At least that will get me away from the FUSE-mount.

OpenStackGossin gravatar imageOpenStackGossin ( 2015-05-21 15:09:16 -0500 )edit