Ask Your Question

How is the implementation of native glusterfs in cinder?

asked 2013-12-11 21:03:59 -0500

edmv gravatar image

Hello there,

I've set up a multi-node environment of openstack using packstack for the company I work for, the main idea behind it is to test it and see the benefit of it for our internal IT operations.

The multi-node setup is made up of 4 servers, 1 that acts as the controller, database, message broker, neutron server, image server and storage, and 3 nodes running nova-compute services and openvswitch agents.

All the hosts are fedora 19 up to date because we wanted to test glusterfs native integration with cinder (libgfapi)

Install/deploy method was packstack and went without issues.

Currently there's only one cinder volume created without any particularities, basically gluster vol create GlusterCinder0 replica 3 vm1:/gluster/vol0 vm2:/gluster/vol0 vm3:/gluster/vol0

The packstack answer file was configured to use gluster and the gluster volume address

Everything went just fine with one exception, I don't see the native implementation in play

After googling I read that I needed to activate the feature by editing qemu_allowed_storage_drivers in nova.conf which I did (restarted cinder-volume and nova services and even all the hosts)

I checked the cinder_shares files which seems ok and currently cinder+glusterfs is working (I can create volumes, attach them to instances, delete volumes, etc...) what seems not to be working is the "native" implementation, by that I mean: I execute "mount" on all the servers and all the mount points used by cinder show fuse.glusterfs as mount type

I can create images directly in the gluster volume using qemu-img create and using native gluster backend (e.g: gluster://ip/vol/image size) so I know that on libvirt side everything seems to work as expected (qemu is version 1.4+, gluster is 3.4.1)

Could you point me to what could I be missing on openstack's side? Maybe I'm not seeing where I should or fuse.glusterfs as mount type is somehow normal, or some better test I could use to verify gluster usage by cinder... anything will be appreciated.

That's all, thanks for reading and sorry for the long post, just wanted to made it clear.

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2013-12-12 23:59:10 -0500

jtopjian gravatar image


I just went through this earlier this week. Indeed, the implementation is a little confusing.

The native implementation is used when booting from a volume only.

Cinder is set up with Gluster just as it was prior to libgfapi, so you'll see the volume mounted under /var/lib/cinder/mnt/abcd via fuse.glusterfs. You use OpenStack just like normal to create both data volumes and bootable volumes.

The difference is that when you go to boot a bootable volume, OpenStack will tell libvirt to use gluster://xxx/vol/uuid as the disk. Now libgfapi is in use.

OpenStack/Nova creates the gluster://xxx/vol/uuid address from your Cinder shares config, so there's not really any magic going on there.

You can verify that the instance was launched using libgfapi by either grepping for gluster in /var/log/nova/nova-compute.log or /var/log/libvirt/qemu/instance-00000000.xml.

Also make sure to check out the "Known Issues" section of the Gluster 3.4 release notes as it has some required settings:

In anticipation that you might run into the same issue that I'm currently stuck at, see this:

I hope that helps!


edit flag offensive delete link more


thanks a ton! I guess it make sense since creating an instance whos image comes from glance doesn't run on cinder, only volume and snapshots. Do you know if this is actually described in any of the docs? I mean, the "only when booting an instance from a volume" part. Thanks again

edmv gravatar imageedmv ( 2013-12-14 08:16:44 -0500 )edit

It's not in the docs yet. There's an existing bug report to have it added:

jtopjian gravatar imagejtopjian ( 2013-12-14 11:33:07 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2013-12-11 21:03:59 -0500

Seen: 1,000 times

Last updated: Dec 12 '13