Ask Your Question
0

Cinder Volume service is down on Controller Node

asked 2016-06-06 01:08:21 -0600

rastogisac gravatar image

updated 2016-06-06 01:12:17 -0600

Dear OpenStack Cinder Experts,

Need your help on solving one issue related to Cinder Service where following is the scenario. I am using three Node configuration for deploying Openstack Liberty release on RHEL OS. I am not able to start Cinder-volume Service on Storage node and also while debugging found that " it is not able to initialize Driver" and fails . given below is the Controleller and storage log. Please advise.

Cinder-Scheduler.log (Controller Node)

2016-06-01 20:23:41.760 24572 CRITICAL cinder [req-2b9c0274-a846-4767-9865-b55b337cc60d - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError)
 (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.rpc_available_version AS services_rpc_available_version, services.object_current_version AS services_object_current_version, services.object_available_version AS services_object_available_version \nFROM services \nWHERE services.deleted = false AND services.topic = %s AND services.disabled = false'] [parameters: ('cinder-volume',)]
2016-06-01 20:23:41.760 24572 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.rpc_available_version AS services_rpc_available_version, services.object_current_version AS services_object_current_version, services.object_available_version AS services_object_available_version \nFROM services \nWHERE services.deleted = false AND services.topic = %s AND services.disabled = false'] [parameters: ('cinder-volume',)]

Cinder-volume log


 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2239
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager [req-357b18bf-274c-4a8e-a65e-54f8362cd542 - - - - -] Failed to initialize driver.
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Traceback (most recent call last):
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 368, in init_host
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     self.driver.check_for_setup_error()
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     return f(*args, **kwargs)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 269, in check_for_setup_error
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     lvm_conf=lvm_conf_file)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 86, in __init__
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     if self._vg_exists() is False:
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line ...
(more)
edit retag flag offensive close merge delete

Comments

According with this error:

2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Stderr: u'  Volume group "cinder-volumes" not found\n  Cannot process volume group cinder-volumes\n'

Do you have a cinder-volume group created? Check it with vgs command

Eduardo Gonzalez gravatar imageEduardo Gonzalez ( 2016-06-06 01:14:25 -0600 )edit

Thanks Edu, Given below is the output [root@pun01cloud03 ~]# vgs cinder-volumes Logging initialised at Mon Jun 6 17:21:02 2016 Using volume group(s) on command line. VG #PV #LV #SN Attr VSize VFree cinder-volumes 1 0 0 wz--n- 1.09t 1.09t Wiping internal VG cache

rastogisac gravatar imagerastogisac ( 2016-06-06 01:20:59 -0600 )edit

3 answers

Sort by ยป oldest newest most voted
0

answered 2016-06-08 03:29:25 -0600

vishmule gravatar image

Ensure cinder's backing storage has enough free space and then restart the cinder-volume and cinder-scheduler services.

# systemctl restart openstack-cinder-volume
# systemctl restart openstack-cinder-scheduler
edit flag offensive delete link more
0

answered 2016-06-06 01:34:10 -0600

rastogisac gravatar image

updated 2016-06-06 01:34:37 -0600

Some more information :

Disk /dev/sdk: 599.6 GB, 599550590976 bytes, 1170997248 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000079c8

Device Boot Start End Blocks Id System /dev/sdk1 2048 398460927 199229440 83 Linux /dev/sdk2 * 398460928 608176127 104857600 83 Linux /dev/sdk3 608176128 796919807 94371840 83 Linux /dev/sdk4 796919808 1170997247 187038720 5 Extended /dev/sdk5 796921856 931139583 67108864 82 Linux swap / Solaris /dev/sdk6 931141632 1119885311 94371840 83 Linux

Disk /dev/sda: 1200.2 GB, 1200243695616 bytes, 2344225968 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdb: 1200.2 GB, 1200243695616 bytes, 2344225968 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdh: 1200.2 GB, 1200243695616 bytes, 2344225968 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@pun01cloud03 ~]# pvdisplay Logging initialised at Mon Jun 6 17:32:54 2016 Using physical volume(s) on command line. Found same device /dev/sdb with same pvid 3MsYgn5YJv6IbzUXrcnsdw0qKSZanZaw --- Physical volume --- PV Name /dev/sdb VG Name cinder-volumes PV Size 1.09 TiB / not usable 1.59 MiB Allocatable yes PE Size 4.00 MiB Total PE 286160 Free PE 286160 Allocated PE 0 PV UUID 3MsYgn-5YJv-6Ibz-UXrc-nsdw-0qKS-ZanZaw

Wiping internal VG cache


[root@pun01cloud03 ~]# vgdisplay
Logging initialised at Mon Jun  6 17:33:41 2016
Using volume group(s) on command line.

--- Volume group --- VG Name cinder-volumes System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 1.09 TiB PE Size 4.00 MiB Total PE 286160 Alloc PE / Size 0 / 0 Free PE / Size 286160 / 1.09 TiB VG UUID G5KKQy-DdCb-m2JQ-4MHb-Iu6W-71e8-txbE3Q

Wiping internal VG cache

[root@pun01cloud03 ~]# grep filter /etc/lvm/lvm.conf |grep -v "#" filter = [ "a/sdb/", "r/.*/"]

edit flag offensive delete link more

Comments

Se

cinder service-list +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+--------

rastogisac gravatar imagerastogisac ( 2016-06-06 05:19:09 -0600 )edit
0

answered 2016-11-18 08:44:45 -0600

Mutty Putty gravatar image

Hi everyone,

           I got the same error, "cinder volume is down". if I restart service, it is showing up for few second then again goes to down state.

here is my cinder-volume.log

2016-11-18 19:00:26.629 2245 CRITICAL cinder [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder Traceback (most recent call last):

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>

2016-11-18 19:00:26.629 2245 ERROR cinder sys.exit(main())

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 109, in main

2016-11-18 19:00:26.629 2245 ERROR cinder launcher.launch_service(server)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 520, in launch_service

2016-11-18 19:00:26.629 2245 ERROR cinder self._start_child(wrap)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 494, in _start_child

2016-11-18 19:00:26.629 2245 ERROR cinder self.launcher.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 236, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder self.services.wait() 2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 687, in wait 2016-11-18 19:00:26.629 2245 ERROR cinder service.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 419, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.rpcserver.wait()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 268, in wrapper

2016-11-18 19:00:26.629 2245 ERROR cinder log_after, timeout_timer)

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 188, in run_once

2016-11-18 19:00:26.629 2245 ERROR cinder post_fn = fn()

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2016-11-18 19:00:26.629 2245 ERROR cinder states[state].run_once(lambda: fn(self, args, *kwargs),

2016-11-18 19:00:26.629 2245 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_messaging/server.py", line 452, in wait

2016-11-18 19:00:26.629 2245 ERROR cinder self.listener.cleanup()

2016-11-18 19:00:26.629 2245 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup'

2016-11-18 19:00:26.629 2245 ERROR cinder

2016-11-18 19:00:26.636 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Caught SIGTERM, stopping children

2016-11-18 19:00:26.642 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Waiting on 1 children to exit

2016-11-18 19:00:26.651 2230 INFO oslo_service.service [req-8c979f13-7cc1-4e02-81ec-0ae8dc16e097 - - - - -] Child 2245 killed by signal 15

2016-11-18 19:00:28.207 3310 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers

SIGUSR1 and SIGUSR2 by default for ... (more)

edit flag offensive delete link more

Comments

1

Yours may be a time issue or rabbitmq. check both if you can. on both controller and cinder nodes.

bcollins gravatar imagebcollins ( 2016-11-18 08:53:22 -0600 )edit

it's not a time issue

here is output

chronyc sources

210 Number of sources = 1

MS Name/IP address Stratum Poll Reach LastRx Last sample

^? controller 0 7 0 10y +0ns[ +0ns] +/-

Mutty Putty gravatar imageMutty Putty ( 2016-11-19 10:28:49 -0600 )edit

root@hpc-cluster-host3:/home/hpchost3# pvdisplay

--- Physical volume --

PV Name /dev/sda4 VG Name cinder-volumes

PV Size 725.70 GiB / not usable 4.57 MiB Allocatable yes PE Size 4.00 MiB Total PE 185777

Mutty Putty gravatar imageMutty Putty ( 2016-11-19 10:30:37 -0600 )edit

When I restart cinder related services on both controller and storage node.

Initially cinder-volume is showing up for few seconds..

After that it is going down..

I think, it is related to thread starting.. Please help me, I am stuck.

@bcollins, thanks.

Mutty Putty gravatar imageMutty Putty ( 2016-11-19 10:44:32 -0600 )edit

Same , issue, someone help

TijoV gravatar imageTijoV ( 2017-11-20 00:21:03 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-06-06 01:08:21 -0600

Seen: 4,389 times

Last updated: Nov 18 '16