cinder volume ignores rabbitmq settings
Hi,
just in the process of setting up cinder and cinder-volume on the storage nodes is ignoring the rabbit settings:
/etc/cinder/cinder.conf
[DEFAULT]
rpc_backend = rabbit
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = secretpassword
Error message:
2016-05-23 15:17:31.014 6100 INFO cinder.volume.manager [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] Updating volume status
2016-05-23 15:17:31.082 6100 ERROR oslo.messaging._drivers.impl_rabbit [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2016-05-23 15:17:32.083 6100 INFO oslo.messaging._drivers.impl_rabbit [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] Reconnecting to AMQP server on localhost:5672
2016-05-23 15:17:32.085 6100 INFO oslo.messaging._drivers.impl_rabbit [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] Delaying reconnect for 1.0 seconds...
2016-05-23 15:17:33.111 6100 ERROR oslo.messaging._drivers.impl_rabbit [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2016-05-23 15:17:36.112 6100 INFO oslo.messaging._drivers.impl_rabbit [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] Reconnecting to AMQP server on localhost:5672
2016-05-23 15:17:36.112 6100 INFO oslo.messaging._drivers.impl_rabbit [req-9e7f0954-9dca-44fb-9728-20d3774652dd - - - - -] Delaying reconnect for 1.0 seconds...
Any idea why its ignoring the settings? Network connectivity is working, rabbitmq is reachable... Running on Ubuntu 14.04 with liberty:
dpkg -l |grep cinder-volume
ii cinder-volume 1:2014.1.5-0ubuntu2 all Cinder storage service - Volume server
UPDATE
When I hack my rabbitmq connection into /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py its working.
When i am creating a image now, I get following error (not sure if this is related):
2016-05-23 17:22:04.531 28011 INFO cinder.volume.manager [req-e8da2747-b860-4bd2-ba68-d7b458eb7898 - - - - -] Starting volume driver RBDDriver (1.1.0)
2016-05-23 17:22:05.314 28011 INFO cinder.volume.manager [req-e8da2747-b860-4bd2-ba68-d7b458eb7898 - - - - -] Updating volume status
2016-05-23 17:22:05.391 28011 INFO oslo.messaging._drivers.impl_rabbit [req-e8da2747-b860-4bd2-ba68-d7b458eb7898 - - - - -] Connected to AMQP server on controller:5672
2016-05-23 17:22:05.457 28011 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on controller:5672
2016-05-23 17:22:37.513 28011 INFO cinder.volume.manager [req-f9ab387f-b502-4207-8433-1710ccebe91e - - - - -] Updating volume status
2016-05-23 17:23:37.513 28011 INFO cinder.volume.manager [-] Updating volume status
2016-05-23 17:24:37.516 28011 INFO cinder.volume.manager [-] Updating volume status
2016-05-23 17:25:37.513 28011 INFO cinder.volume.manager [-] Updating volume status
2016-05-23 17:26:37.514 28011 INFO cinder.volume.manager [-] Updating volume status
2016-05-23 17:27:37.514 28011 INFO cinder.volume.manager [-] Updating volume status
2016-05-23 17:27:46.174 28011 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Endpoint does not support RPC version 1.24
2016-05-23 17:27:46.174 28011 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-23 17:27:46.174 28011 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2016-05-23 17:27:46.174 28011 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2016-05-23 17:27:46.174 28011 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py ...
SOLUTION: the storage node and the controller node hat two different versions running.