Ask Your Question
0

volume service is down

asked 2016-01-03 01:49:46 -0500

matansab gravatar image

updated 2016-01-03 22:56:46 -0500

Bipin gravatar image

All of a sudden I cannot perform any action on cinder (like create for example)

I ran cinder-manage service list and got

Binary           Host                                 Zone             Status     State Updated At

cinder-backup    localhost                            nova             enabled    :-)   2015-12-31 06:30:04

cinder-scheduler localhost                            nova             enabled    :-)   2015-12-31 06:30:05

cinder-volume    localhost                            nova             enabled    XXX   2015-12-13 21:47:37

cinder-volume    localhost                            nova             enabled    :-)   2015-12-31 06:30:05

and also service openstack-cinder-volume status

Redirecting to /bin/systemctl status  openstack-cinder-volume.service

openstack-cinder-volume.service - OpenStack Cinder Volume Server

   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-volume.service; enabled)

   Active: active (running) since Thu 2015-12-31 01:03:56 EST; 26min ago

I tried to create a volume just in case and saw under cinder’s scheduler.log :

2015-12-31 01:25:41.098 1052 ERROR cinder.scheduler.flows.create_volume [req-9e5a2373-b06b-4320-9293-05d3f7d0638d 34f31b50e2554f68b55419d6de5a8e25 5aa43c3e3e3c4558a982c99542b44248 - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid host was found. No weighed hosts available

Other logs that indicates the same:

-98dbce07a7b9 - - - - -] volume service is down. (host: localhost.localdomain@lvm)
2015-12-31 01:04:43.721 1052 WARNING cinder.scheduler.host_manager [req-edb5abbd-9405-4511-821b-98dbce07a7b9 - - - - -] volume service is down. (host: localhost.localdomain@scaleio)
2015-12-31 01:04:43.791 1052 WARNING oslo_config.cfg [req-edb5abbd-9405-4511-821b-98dbce07a7b9 - - - - -] Option "amqp_durable_queues" from group "DEFAULT" is deprecated. Use option "amqp_durable_queues" from group "oslo_messaging_rabbit".
2015-12-31 01:04:43.794 1052 WARNING oslo_config.cfg [req-edb5abbd-9405-4511-821b-98dbce07a7b9 - - - - -] Option "lock_path" from group "DEFAULT" is deprecated. Use option "lock_path" from group "oslo_concurrency".
2015-12-31 01:04:43.795 1052 INFO cinder.service [-] Starting cinder-scheduler node (version 7.0.0)
2015-12-31 01:04:45.215 1052 INFO oslo.messaging._drivers.impl_rabbit [req-bfa51c3d-471b-4a5c-a001-b1d2fce90567 - - - - -] Connecting to AMQP server on 10.136.242.238:5672
2015-12-31 01:11:55.414 1052 WARNING cinder.scheduler.host_manager [req-bb076884-9d67-40d9-bbd8-49eaf9b97fa6 34f31b50e2554f68b55419d6de5a8e25 5aa43c3e3e3c4558a982c99542b44248 - - -] volume service is down. (host: localhost.localdomain@lvm)
2015-12-31 01:11:55.416 1052 INFO cinder.openstack.common.scheduler.base_filter [req-bb076884-9d67-40d9-bbd8-49eaf9b97fa6 34f31b50e2554f68b55419d6de5a8e25 5aa43c3e3e3c4558a982c99542b44248 - - -] Filter CapabilitiesFilter returned 0 host(s)
2015-12-31 01:11:55.417 1052 ERROR cinder.scheduler.manager [req-bb076884-9d67-40d9-bbd8-49eaf9b97fa6 34f31b50e2554f68b55419d6de5a8e25 5aa43c3e3e3c4558a982c99542b44248 - - -] Could
edit retag flag offensive close merge delete

Comments

Make sure, your rabbitmq server is up and able to connect from Cinder.

Bipin gravatar imageBipin ( 2016-01-03 22:59:07 -0500 )edit

2 answers

Sort by » oldest newest most voted
0

answered 2016-11-18 08:28:28 -0500

Mutty Putty gravatar image

updated 2016-11-18 08:32:08 -0500

Hi everyone,

           I got the same error, "cinder volume is down". if I restart service, it is showing up for few second then again goes to down state.

here is my cinder-volume.log

2016-11-18 19:00:26.629 2245 CRITICAL cinder [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder Traceback (most recent call last):

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>

2016-11-18 19:00:26.629 2245 ERROR cinder sys.exit(main())

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 109, in main

2016-11-18 19:00:26.629 2245 ERROR cinder launcher.launch_service(server)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 520, in launch_service

2016-11-18 19:00:26.629 2245 ERROR cinder self._start_child(wrap)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 494, in _start_child

2016-11-18 19:00:26.629 2245 ERROR cinder self.launcher.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 236, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder self.services.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 687, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder service.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 419, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.rpcserver.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:26.629 2245 ERROR cinder log_after, timeout_timer)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:26.629 2245 ERROR cinder post_fn = fn()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:26.629 2245 ERROR cinder states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 452, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.listener.cleanup()

2016-11-18 19:00:26.629 2245 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder

2016-11-18 19:00:26.636 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Caught SIGTERM, stopping children

2016-11-18 19:00:26.642 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Waiting on 1 children to exit

2016-11-18 19:00:26.651 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Child 2245 killed by signal 15

2016-11-18 19:00:28.207 3310 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers

SIGUSR1 and SIGUSR2 by default for ... (more)

edit flag offensive delete link more
0

answered 2016-03-14 09:32:40 -0500

mao18 gravatar image
Volume service is down,maybe there is something wrong with your ntp service.Try to check the time of your controler and cinder node to insure they are the same.Cinder list tell you the cinder-volume service is down,so you can try to check the code of it.
class ServiceController(wsgi.Controller):
@wsgi.serializers(xml=ServicesIndexTemplate)
def index(self, req):
    """Return a list of all running services.
    Filter by host & service name.
    """
    context = req.environ['cinder.context']
    authorize(context)
    detailed = self.ext_mgr.is_loaded('os-extended-services')
    now = timeutils.utcnow() //fetch controller time
    services = db.service_get_all(context) // get cinder service from db
    ...
    svcs = []
    for svc in services: //for each service
        delta = now - (svc['updated_at'] or svc['created_at']) //get updated_at。if not, created_at,and compute time difference
        alive = abs(utils.total_seconds(delta)) <= CONF.service_down_time //get time difference and compare with server_down_time,(60s default)
        art = (alive and "up") or "down" ***//if lease than 60s ,up state, otherwise down state.**
        active = 'enabled'
        ......
        svcs.append(ret_fields)
    return {'services': svcs}

reference [link text](http://www.cnblogs.com/sammyliu/p/4417091.html)
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-01-03 01:49:46 -0500

Seen: 2,592 times

Last updated: Nov 18 '16