Unable to use ceph for nova instance storage in newton

asked 2016-12-12 10:28:05 -0500

proceonmw gravatar image

updated 2016-12-12 15:40:34 -0500

rbowen gravatar image

Greetings -

Having trouble getting nova to use ceph for instance storage...glance images and cinder volumes are working fine with ceph. Have been generally following these instructions: http://www.stratoscale.com/blog/stora.... Everything including nova works great in Mitaka, but no luck with Newton for nova. Nova insists on using LVM under iscsi on the cinder host for instance storage. Have searched to see if there have been config changes made in Newton that would affect this? Not getting any errors in nova-compute.log, just instance will always get created on LVM/cinder host....I am using glance images off of ceph as an image for instance.

On compute node:

General /etc/nova/nova.conf entries:
[libvirt]
images_type=rbd <---- and have used #images_type=default (no luck)
images_rbd_pool = newton
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = compute
rbd_secret_uuid = XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX

Added to /etc/ceph/ceph.conf:
[client.compute]
keyring = /etc/ceph/ceph.client.compute.keyring

Defined secret for libvirt etc.created users under ceph, copied keys etc..basically same steps as done with Mitaka, no dice with Newton.

Thanks for any help.

edit retag flag offensive close merge delete

Comments

If debug logging is enabled on that compute node, the nova-compute log contains the actual values of config parameters when nova-compute starts, and should also have some messages about setting up instance storage that might help you understand what's happening.

Bernd Bausch gravatar imageBernd Bausch ( 2016-12-12 18:35:03 -0500 )edit

Yep...have debug logging turned on and I do see the rbd params going by....libvirt.images_rbd_ceph_conf, libvirt.images_rbd_pool...etc with their corresponding values, but no calls to rbd_utils like in Mitaka. Every instance image goes straight to iscsi. Anyone have steps for nova/ceph on newton?

proceonmw gravatar imageproceonmw ( 2016-12-13 10:42:38 -0500 )edit

Thanks for the help....still no success. Is there a away to actively query the backend storage that nova will use when spinning up an instance? Perhaps thru API? I have looked at the debug output from Mitaka and Newton, and Newton only lists my libvirt.rbd values, but does not switch to using RBD

proceonmw gravatar imageproceonmw ( 2016-12-19 14:56:52 -0500 )edit

Update...I originally had this issue with Centos7 and Nova 14.0.0 (Newton) on two separate clusters. Installed Ubuntu 16.04 with Nova 14.0.1...no problems was able to use ceph with Nova. Reinstalled with Centos7 and Nova 14.0.2 and that corrected the problem...no issues now.....Thx for the help

proceonmw gravatar imageproceonmw ( 2017-01-19 08:07:59 -0500 )edit