Unable to integrate openstack with ceph

asked 2018-04-04 08:22:22 -0500

Vikas M gravatar image

updated 2018-04-04 08:34:19 -0500

I have installed 5 node ceph( ceph version 10.2.10) setup ( 1 ceph-admin + 1 mon + 3 osd nodes ) and prepared one node openstack pike(packstack).

I am trying to integrate ceph with openstack , All steps executed with out error But when I run #rbd ls images OR #rbd -p images ls It is showing no output.

[root@ceph-admin ~]# ceph osd pool ls
[root@ceph-admin ~]#
[root@ceph-admin ~]# rbd ls images
[root@ceph-admin ~]#

I executed below commands to integrate ceph with openstack

    1. #ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
    2. #ceph auth get-or-create client.glance | ssh controller sudo tee /etc/ceph/ceph.client.glance.keyring
    3. #ssh controller chown glance:glance /etc/ceph/ceph.client.glance.keyring
    4.  edited ceph.conf on openstack node as per below

    [root@controller ~]# cat /etc/ceph/ceph.conf
    fsid = 91336825-14db-4daf-ad42-ccf29e598b2b
    mon_initial_members = mon1
    mon_host =
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx

    public network =
    osd pool default size = 2

    keyring = /etc/ceph/ceph.client.glance.keyring
    [root@controller ~]#

   5. added below entries in /etc/glance/glance-api.conf 

    stores = rbd
    rbd_store_pool = images
    rbd_store_user = glance
    rbd_store_ceph_conf = /etc/ceph/ceph.conf
    rbd_store_chunk_size = 8
edit retag flag offensive close merge delete


Did you upload any images?

I think you also need to set the default_store value in glance-api.conf. If not, a default of fileis assumed, and any images are stored in /var/lib/glance/images. See https://docs.openstack.org/glance/lat....

Bernd Bausch gravatar imageBernd Bausch ( 2018-04-04 08:55:23 -0500 )edit

1 answer

Sort by » oldest newest most voted

answered 2018-04-04 09:29:28 -0500

Vikas M gravatar image

Yes I have tried to upload image many times. It is uploading without error. But If I change stores = rbd,file,http,swift to stores = rbd in glance-api.conf and try to upload the image it is showing below error.

[root@controller ~]# openstack image create --file cirros-0.3.4-x86_64-disk.raw --container-format bare --disk-format raw --public cirros-0.3.4.ceph

Error finding address for HTTPConnectionPool(host='', port=9292): Max retries exceeded with url: /v2/schemas/image (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3a2c4d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

[root@controller ~]#

edit flag offensive delete link more


Connection refused indicates that glance-api is not running on Perhaps crashed. What do you see in the Glance logs?

Bernd Bausch gravatar imageBernd Bausch ( 2018-04-04 09:44:56 -0500 )edit

When I revert back to stores = rbd,file,http,swift , Glance works fine and port 9292 also listen

[root@controller ~]# netstat -anp |grep 9292
tcp        0      0  *               LISTEN      22674/python2

But when I change it to stores = rbd It stops workin

Vikas M gravatar imageVikas M ( 2018-04-04 10:44:33 -0500 )edit

When you set stores = rdb, does glance-api run? If no, check the glance-api log for information why it’s stopped. If yes, use network tools to find out why it’s not reachable, and also check the log.

Bernd Bausch gravatar imageBernd Bausch ( 2018-04-04 17:25:04 -0500 )edit

When you set stores = rbd,file,http,swift and default_store = rdb, what happens?

Bernd Bausch gravatar imageBernd Bausch ( 2018-04-04 17:26:32 -0500 )edit

How is your network design? Are you using different network interface/vlan for data network? Is the network between controller and ceph nodes OK? Can they ping each other from their data interface? If it is ok, you can check ceph nodes health by using ceph -s command.

onderogluserdar gravatar imageonderogluserdar ( 2018-04-12 13:24:27 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2018-04-04 08:22:22 -0500

Seen: 339 times

Last updated: Apr 04 '18