Ceph mon on neutron controller node

asked 2020-02-19 09:10:52 -0500

Samy Ascha gravatar image


I have a setup where my OpenStack Stein controller/neutron node are also running Ceph monitor daemons.

Since I have only limited NICs available, I have 2 bonds, with the following functions:

  • 10Gb/s, used for provider network / internet access
  • 40Gb/s, used for MySQL, RabbitMQ and Ceph public network AND tenant networking with vxlan / linux bridge

I'm using a flat network to expose my Ceph cluster to OpenStack instances. This results in a situation where I have neutron nodes with a static IP in combination with a flat network that, afaik, should not have an IP assigned.

The setup seems to work OK, as far as running instances, tenant network, and using ceph storage goes. However, i feel this is not the way it should be setup. For example, I see the IP for this host being assigned to a bridge (brqX) interface.

I have had some occasions where, although I have a HA setup with 3 controller nodes, bringing down a specific one took down all tenant networking for a specific project.

Do any of you have general advice or, for that matter, string objections against such a setup?

How would I best go about offering Ceph public storage access to instances AND OpenStack services AND vxlan, when I have only 1 physical NIC available for these services. In this environment I'm not worried about bandwidth. It seems more than enough.

I'm happy to provide more details, explanation, configs and logs.

Thx very much!


edit retag flag offensive close merge delete


To be honest, I would recommend to separate ceph from openstack and rather deploy MON services on OSD nodes. MON daemons don't require much CPU and RAM. That way ceph is not affected if you have to maintain the control node(s) or if they fail.

eblock gravatar imageeblock ( 2020-02-25 03:11:38 -0500 )edit