can I add an OSD to ceph that is nfs mount

asked 2017-01-18 16:08:53 -0600

jamesopst gravatar image

I have CEPH configured for CINDER AND GLANCE on a storage node in an openstack deployment

Until now both CINDER AND GLANCE have been using a fiber-channel storage device connected to the storage node

Now I would like to move GLANCE off the fiber-channel device and onto a standard (slower) network share, nfs.

Is it possible to use a nfs mount with CEPH?

I assume I need to start by adding an OSD using the nfs mounted directory as the filestore. This doesn't seem to work (see errors below), but I am sure I jumped the gun and need to enable CEPH/nfs support in some way. I am searching for a tutorial about this but I keep coming up with exportinhg CEPH-FS over NFS. Which is very interesting, but not what I need exactly. thanks for any advice you can offer!

2017-01-18 21:46:46.210822 7f26b8ece800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2017-01-18 21:46:46.237724 7f26b8ece800 -1 filestore(/var/lib/ceph/osd/ceph-6) Extended attributes don't appear to work. Got error (95) Operation not supported. If you are using ext3 or ext4, be sure to mount the underlying file system with the 'user_xattr' option.
2017-01-18 21:46:46.238520 7f26b8ece800 -1 filestore(/var/lib/ceph/osd/ceph-6) FileStore::mount : error in _detect_fs: (95) Operation not supported
2017-01-18 21:46:46.238776 7f26b8ece800 -1 OSD::mkfs: couldn't mount ObjectStore: error -95
2017-01-18 21:46:46.238811 7f26b8ece800 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-6: (95) Operation not supported
edit retag flag offensive close merge delete


What do you mean by "move Glance to a network share, NFS"? You could configure a local filesystem as Glance store, and mount that "local" filesystem from an NFS server. When you use Ceph, you use the network anyway. But "NFS mount with Ceph" - can you be more precise what you mean?

Bernd Bausch gravatar imageBernd Bausch ( 2017-01-18 17:55:03 -0600 )edit

I think I understand - /var/lib/ceph/osd/ceph-123 as NFS mount. The error message is clear - NFS doesn't support extended attributes. See also But even if it can be made to work, does it make sense?

Bernd Bausch gravatar imageBernd Bausch ( 2017-01-18 17:58:40 -0600 )edit

thank you for your reply. from what I it looks like it can't easily be made to work... right? too bad. but as you suggest, does it make sense? admittedly I am in over my head here...

jamesopst gravatar imagejamesopst ( 2017-01-19 11:26:11 -0600 )edit

but to explain: The objective of my task here is to move image storage (Glance) to another file system so that the current local file system (fast fiber-channel) can be all for block storage for running instances (Cinder). Could you please comment on that if it makes sense? thanks!

jamesopst gravatar imagejamesopst ( 2017-01-19 11:29:42 -0600 )edit

Sure, if you don't want to occupy your expensive fast storage with images, move them somewhere else. But you don't need Ceph to achieve that. Just set up another image store (even Cinder can be used), copy the images and unconfigure the first store.

Bernd Bausch gravatar imageBernd Bausch ( 2017-01-20 07:59:05 -0600 )edit