Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Hi everyone, I get the same error, "cinder volume is down". if I restart service, it is show up for few second then again goes to down state.

here is my cinder-volume.log

2016-11-18 19:00:26.629 2245 CRITICAL cinder [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder Traceback (most recent call last):

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>

2016-11-18 19:00:26.629 2245 ERROR cinder sys.exit(main())

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 109, in main

2016-11-18 19:00:26.629 2245 ERROR cinder launcher.launch_service(server)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 520, in launch_service

2016-11-18 19:00:26.629 2245 ERROR cinder self._start_child(wrap)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 494, in _start_child

2016-11-18 19:00:26.629 2245 ERROR cinder self.launcher.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 236, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder self.services.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 687, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder service.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 419, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.rpcserver.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:26.629 2245 ERROR cinder log_after, timeout_timer)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:26.629 2245 ERROR cinder post_fn = fn()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:26.629 2245 ERROR cinder states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 452, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.listener.cleanup()

2016-11-18 19:00:26.629 2245 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder

2016-11-18 19:00:26.636 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Caught SIGTERM, stopping children

2016-11-18 19:00:26.642 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Waiting on 1 children to exit

2016-11-18 19:00:26.651 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Child 2245 killed by signal 15

2016-11-18 19:00:28.207 3310 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers

SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

2016-11-18 19:00:28.265 3310 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt

2016-11-18 19:00:28.285 3310 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt

2016-11-18 19:00:28.445 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-volume objects version 1.11 as minimum service version.

2016-11-18 19:00:28.457 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-scheduler objects version 1.11 as minimum service version.

2016-11-18 19:00:28.462 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-scheduler RPC version 3.0 as minimum service version.

2016-11-18 19:00:28.517 3310 INFO cinder.volume.manager [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Determined volume DB was empty at startup.

2016-11-18 19:00:28.522 3310 WARNING cinder.keymgr.conf_key_mgr [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] This key manager is insecure and is not recommended for production deployments

2016-11-18 19:00:28.530 3310 INFO cinder.volume.manager [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Image-volume cache disabled for host hpc-cluster-host3@lvm.

2016-11-18 19:00:28.530 3310 INFO oslo_service.service [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Starting 1 workers

2016-11-18 19:00:28.541 3325 INFO cinder.service [-] Starting cinder-volume node (version 9.0.0)

2016-11-18 19:00:28.560 3325 INFO cinder.coordination [-] Coordination backend started successfully.

2016-11-18 19:00:28.561 3325 INFO cinder.volume.manager [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0)

2016-11-18 19:00:31.644 3325 INFO cinder.volume.manager [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Driver initialization completed successfully.

2016-11-18 19:00:31.665 3325 ERROR oslo.messaging._drivers.impl_rabbit [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Failed to declare consumer for topic 'cinder-volume': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'false' but current is 'true'

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Error starting thread.

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service Traceback (most recent call last):

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 708, in run_service

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service service.start()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 247, in start

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.rpcserver.start()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service log_after, timeout_timer)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service post_fn = fn()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 420, in start

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.listener = self._create_listener()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return self.transport._listen(self._target, 1, None)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 112, in _listen

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service batch_timeout)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 477, in listen

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service callback=listener)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1127, in declare_topic_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.declare_consumer(consumer)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1015, in declare_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service error_callback=_connect_error)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service ret, channel = autoretry_method()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return fun(args, *kwargs)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return fun(args, channel=channels[0], *kwargs), channels[0]

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 767, in execute_method

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service method()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1003, in _declare_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service consumer.declare(self)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 300, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.queue.declare()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.exchange.declare(nowait)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service nowait=nowait, passive=passive,

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 619, in exchange_declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service (40, 11), # Channel.exchange_declare_ok

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return self.dispatch_method(method_sig, args, content)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages

/amqp/abstract_channel.py", line 87, in dispatch_method

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return amqp_method(self, args)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 243, in _close

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service reply_code, reply_text, (class_id, method_id), ChannelError,

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'false' but current is 'true'

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service

2016-11-18 19:01:01.670 3325 WARNING oslo_messaging.server [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Possible hang: wait is waiting for stop to complete.

                                                                                                                                                                          Thanks in advance.

Any help will be appreciated.

Hi everyone, everyone,

           I get got the same error, "cinder volume is down". if I restart service, it is show showing up for few second then again goes to down state.

state.

here is my cinder-volume.log

2016-11-18 19:00:26.629 2245 CRITICAL cinder [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder Traceback (most recent call last):

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>

2016-11-18 19:00:26.629 2245 ERROR cinder sys.exit(main())

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 109, in main

2016-11-18 19:00:26.629 2245 ERROR cinder launcher.launch_service(server)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 520, in launch_service

2016-11-18 19:00:26.629 2245 ERROR cinder self._start_child(wrap)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 494, in _start_child

2016-11-18 19:00:26.629 2245 ERROR cinder self.launcher.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 236, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder self.services.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 687, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder service.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 419, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.rpcserver.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:26.629 2245 ERROR cinder log_after, timeout_timer)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:26.629 2245 ERROR cinder post_fn = fn()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:26.629 2245 ERROR cinder states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 452, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.listener.cleanup()

2016-11-18 19:00:26.629 2245 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder

2016-11-18 19:00:26.636 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Caught SIGTERM, stopping children

2016-11-18 19:00:26.642 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Waiting on 1 children to exit

2016-11-18 19:00:26.651 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Child 2245 killed by signal 15

2016-11-18 19:00:28.207 3310 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers

SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

2016-11-18 19:00:28.265 3310 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt

2016-11-18 19:00:28.285 3310 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt

2016-11-18 19:00:28.445 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-volume objects version 1.11 as minimum service version.

2016-11-18 19:00:28.457 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-scheduler objects version 1.11 as minimum service version.

2016-11-18 19:00:28.462 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-scheduler RPC version 3.0 as minimum service version.

2016-11-18 19:00:28.517 3310 INFO cinder.volume.manager [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Determined volume DB was empty at startup.

2016-11-18 19:00:28.522 3310 WARNING cinder.keymgr.conf_key_mgr [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] This key manager is insecure and is not recommended for production deployments

2016-11-18 19:00:28.530 3310 INFO cinder.volume.manager [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Image-volume cache disabled for host hpc-cluster-host3@lvm.

2016-11-18 19:00:28.530 3310 INFO oslo_service.service [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Starting 1 workers

2016-11-18 19:00:28.541 3325 INFO cinder.service [-] Starting cinder-volume node (version 9.0.0)

2016-11-18 19:00:28.560 3325 INFO cinder.coordination [-] Coordination backend started successfully.

2016-11-18 19:00:28.561 3325 INFO cinder.volume.manager [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0)

2016-11-18 19:00:31.644 3325 INFO cinder.volume.manager [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Driver initialization completed successfully.

2016-11-18 19:00:31.665 3325 ERROR oslo.messaging._drivers.impl_rabbit [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Failed to declare consumer for topic 'cinder-volume': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'false' but current is 'true'

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Error starting thread.

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service Traceback (most recent call last):

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 708, in run_service

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service service.start()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 247, in start

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.rpcserver.start()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service log_after, timeout_timer)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service post_fn = fn()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 420, in start

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.listener = self._create_listener()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return self.transport._listen(self._target, 1, None)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 112, in _listen

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service batch_timeout)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 477, in listen

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service callback=listener)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1127, in declare_topic_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.declare_consumer(consumer)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1015, in declare_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service error_callback=_connect_error)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service ret, channel = autoretry_method()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return fun(args, *kwargs)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return fun(args, channel=channels[0], *kwargs), channels[0]

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 767, in execute_method

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service method()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1003, in _declare_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service consumer.declare(self)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 300, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.queue.declare()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.exchange.declare(nowait)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service nowait=nowait, passive=passive,

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 619, in exchange_declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service (40, 11), # Channel.exchange_declare_ok

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return self.dispatch_method(method_sig, args, content)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages

/amqp/abstract_channel.py", line 87, in dispatch_method

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return amqp_method(self, args)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 243, in _close

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service reply_code, reply_text, (class_id, method_id), ChannelError,

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'false' but current is 'true'

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service

2016-11-18 19:01:01.670 3325 WARNING oslo_messaging.server [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Possible hang: wait is waiting for stop to complete.

                                                                                                                                                                          Thanks in advance.

Any help will be appreciated.

Hi everyone,

           I got the same error, "cinder volume is down". if I restart service, it is showing up for few second then again goes to down state.

here is my cinder-volume.log

2016-11-18 19:00:26.629 2245 CRITICAL cinder [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder Traceback (most recent call last):

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>

2016-11-18 19:00:26.629 2245 ERROR cinder sys.exit(main())

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 109, in main

2016-11-18 19:00:26.629 2245 ERROR cinder launcher.launch_service(server)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 520, in launch_service

2016-11-18 19:00:26.629 2245 ERROR cinder self._start_child(wrap)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 494, in _start_child

2016-11-18 19:00:26.629 2245 ERROR cinder self.launcher.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 236, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder self.services.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 687, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder service.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 419, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.rpcserver.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:26.629 2245 ERROR cinder log_after, timeout_timer)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:26.629 2245 ERROR cinder post_fn = fn()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:26.629 2245 ERROR cinder states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 452, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.listener.cleanup()

2016-11-18 19:00:26.629 2245 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder

2016-11-18 19:00:26.636 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Caught SIGTERM, stopping children

2016-11-18 19:00:26.642 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Waiting on 1 children to exit

2016-11-18 19:00:26.651 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Child 2245 killed by signal 15

2016-11-18 19:00:28.207 3310 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers

SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

2016-11-18 19:00:28.265 3310 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt

2016-11-18 19:00:28.285 3310 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt

2016-11-18 19:00:28.445 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-volume objects version 1.11 as minimum service version.

2016-11-18 19:00:28.457 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-scheduler objects version 1.11 as minimum service version.

2016-11-18 19:00:28.462 3310 INFO cinder.rpc [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Automatically selected cinder-scheduler RPC version 3.0 as minimum service version.

2016-11-18 19:00:28.517 3310 INFO cinder.volume.manager [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Determined volume DB was empty at startup.

2016-11-18 19:00:28.522 3310 WARNING cinder.keymgr.conf_key_mgr [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] This key manager is insecure and is not recommended for production deployments

2016-11-18 19:00:28.530 3310 INFO cinder.volume.manager [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Image-volume cache disabled for host hpc-cluster-host3@lvm.

2016-11-18 19:00:28.530 3310 INFO oslo_service.service [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Starting 1 workers

2016-11-18 19:00:28.541 3325 INFO cinder.service [-] Starting cinder-volume node (version 9.0.0)

2016-11-18 19:00:28.560 3325 INFO cinder.coordination [-] Coordination backend started successfully.

2016-11-18 19:00:28.561 3325 INFO cinder.volume.manager [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0)

2016-11-18 19:00:31.644 3325 INFO cinder.volume.manager [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Driver initialization completed successfully.

2016-11-18 19:00:31.665 3325 ERROR oslo.messaging._drivers.impl_rabbit [req-1a12a036-b682-4f33-9e56-97ef0595ea03 oslo.messaging._drivers.impl_rabbit [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Failed to declare consumer for topic 'cinder-volume': 'cinder-volume': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg arg 'durable' for exchange 'openstack' in in vhost '/': received 'false' but but current is 'true'

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service [req-1a12a036-b682-4f33-9e56-97ef0595ea03 - - - - -] Error starting thread.

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service Traceback (most recent call last):

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 708, in run_service

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service service.start()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 247, in start

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.rpcserver.start()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service log_after, timeout_timer)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service post_fn = fn()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 420, in start

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.listener = self._create_listener()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return self.transport._listen(self._target, 1, None)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 112, in _listen

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service batch_timeout)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 477, in listen

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service callback=listener)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1127, in declare_topic_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.declare_consumer(consumer)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1015, in declare_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service error_callback=_connect_error)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service ret, channel = autoretry_method()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return fun(args, *kwargs)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return fun(args, channel=channels[0], *kwargs), channels[0]

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 767, in execute_method

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service method()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1003, in _declare_consumer

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service consumer.declare(self)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 300, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.queue.declare()

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service self.exchange.declare(nowait)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service nowait=nowait, passive=passive,

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 619, in exchange_declare

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service (40, 11), # Channel.exchange_declare_ok

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return self.dispatch_method(method_sig, args, content)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages

/amqp/abstract_channel.py", line 87, in dispatch_method

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service return amqp_method(self, args)

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 243, in _close

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service reply_code, reply_text, (class_id, method_id), ChannelError,

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service oslo_service.service PreconditionFailed: Exchange.declare: Exchange.declare: (406) PRECONDITION_FAILED - - inequivalent arg 'durable' for for exchange 'openstack' in vhost '/': '/': received 'false' but current is is 'true'

2016-11-18 19:00:31.667 3325 ERROR oslo_service.service

2016-11-18 19:01:01.670 3325 WARNING oslo_messaging.server [req-1639d2d2-8c45-494b-847a-c9c2069b1ee3 - - - - -] Possible hang: wait is waiting for stop to complete.

                                                                                                                                                                          Thanks in advance.

Any help will be appreciated.