Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Cinder Volume service is down on Controller Node

Dear OpenStack Cinder Experts,

Need your help on solving one issue related to Cinder Service where following is the scenario. I am using three Node configuration for deploying Openstack Liberty release on RHEL OS. I am not able to start Cinder-volume Service on Storage node and also while debugging found that " it is not able to initialize Driver" and fails . given below is the Controleller and storage log. Please advise.

Cinder-Scheduler.log (Controller Node) 2016-06-01 20:23:41.760 24572 CRITICAL cinder [req-2b9c0274-a846-4767-9865-b55b337cc60d - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.binary AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.rpc_available_version AS services_rpc_available_version, services.object_current_version AS services_object_current_version, services.object_available_version AS services_object_available_version \nFROM services \nWHERE services.deleted = false AND services.topic = %s AND services.disabled = false'] [parameters: ('cinder-volume',)] 2016-06-01 20:23:41.760 24572 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.binary AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.rpc_available_version AS services_rpc_available_version, services.object_current_version AS services_object_current_version, services.object_available_version AS services_object_available_version \nFROM services \nWHERE services.deleted = false AND services.topic = %s AND services.disabled = false'] [parameters: ('cinder-volume',)]

Cinder-volume log ****** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2239 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager [req-357b18bf-274c-4a8e-a65e-54f8362cd542 - - - - -] Failed to initialize driver. 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Traceback (most recent call last): 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 368, in init_host 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager self.driver.check_for_setup_error() 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager return f(args, *kwargs) 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 269, in check_for_setup_error 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager lvm_conf=lvm_conf_file) 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 86, in __init__ 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager if self._vg_exists() is False: 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 123, in _vg_exists 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager run_as_root=True) 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 155, in execute 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager return processutils.execute(cmd, **kwargs) 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager cmd=sanitized_cmd) 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command. 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Command: env LC_ALL=C vgs --noheadings -o name cinder-volumes 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Exit code: 5 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Stdout: u'' 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Stderr: u' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n' 2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager 2016-06-03 23:12:02.895 3171 DEBUG oslo_db.sqlalchemy.engines [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Parent process 3152 forked (3171) with an open database connection, which is being discarded and recreated. checkout /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:357 2016-06-03 23:12:02.904 3171 DEBUG cinder.service [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Creating RPC server for service cinder-volume start /usr/lib/python2.7/site-packages/cinder/service.py:155 2016-06-03 23:12:02.926 3171 DEBUG oslo_messaging._drivers.amqp [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Pool creating new connection create /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:103 2016-06-03 23:12:02.933 3171 INFO oslo.messaging._drivers.impl_rabbit [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Connecting to AMQP server on 192.168.170.30:5672 2016-06-03 23:12:10.944 3171 INFO oslo.messaging._drivers.impl_rabbit [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Connected to AMQP server on 192.168.170.30:5672 2016-06-03 23:12:20.959 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down". 2016-06-03 23:12:30.963 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down". 2016-06-03 23:12:40.963 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down". 2016-06-03 23:12:50.973 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down". 2016-06-03 23:13:00.978 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down". 2016-06-03 23:13:10.978 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down". 2016-06-03 23:13:11.452 3171 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting 2016-06-03 23:13:11.454 3152 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting

[root@pun01cloud01 cinder]# cinder service-list +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | pun01cloud01.ad.infosys.com | nova | enabled | up | 2016-06-06T11:24:48.000000 | - | | cinder-volume | pun01cloud01.ad.infosys.com | nova | enabled | down | - | - | +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+

Storage Node (cinder-manage) 2016-06-02 21:40:00.253 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d 8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 10 attempts left. 2016-06-02 21:40:10.264 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d 8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 9 attempts left. 2016-06-02 21:40:20.275 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d 8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 8 attempts left. 2016-06-02 21:40:30.285 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d 8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 7 attempts left. 2016-06-02 21:40:33.478 78979 CRITICAL cinder [req-2d407b86-d8ca-4b75-a932-0acf1 e4e65f5 - - - - -] KeyboardInterrupt 2016-06-02 21:40:33.478 78979 ERROR cinder Traceback (most recent call last): 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/bin/cinder-manage", line 10, in <module> 2016-06-02 21:40:33.478 78979 ERROR cinder sys.exit(main()) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/cmd/manage.py", line 592, in main 2016-06-02 21:40:33.478 78979 ERROR cinder fn(fn_args) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/cmd/manage.py", line 439, in list 2016-06-02 21:40:33.478 78979 ERROR cinder services = objects.ServiceList.ge t_all(ctxt) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/oslo_versionedobjects/base.py", line 171, in wrapper 2016-06-02 21:40:33.478 78979 ERROR cinder result = fn(cls, context, *args, *kwargs) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/objects/service.py", line 119, in get_all 2016-06-02 21:40:33.478 78979 ERROR cinder services = db.service_get_all(con text, filters) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/db/api.py", line 110, in service_get_all 2016-06-02 21:40:33.478 78979 ERROR cinder return IMPL.service_get_all(conte xt, disabled) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/db/sqlalchemy/api.py", line 165, in wrapper 2016-06-02 21:40:33.478 78979 ERROR cinder return f(args, *kwargs) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/db/sqlalchemy/api.py", line 365, in service_get_all 2016-06-02 21:40:33.478 78979 ERROR cinder query = model_query(context, mode ls.Service) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/db/sqlalchemy/api.py", line 242, in model_query 2016-06-02 21:40:33.478 78979 ERROR cinder session = kwargs.get('session') o r get_session() 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/db/sqlalchemy/api.py", line 92, in get_session 2016-06-02 21:40:33.478 78979 ERROR cinder facade = _create_facade_lazily() 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/cinder/db/sqlalchemy/api.py", line 74, in _create_facade_lazily 2016-06-02 21:40:33.478 78979 ERROR cinder *dict(CONF.database) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__ 2016-06-02 21:40:33.478 78979 ERROR cinder slave_connection=slave_connection ) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start 2016-06-02 21:40:33.478 78979 ERROR cinder engine_args, maker_args) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection 2016-06-02 21:40:33.478 78979 ERROR cinder sql_connection=sql_connection, * engine_kwargs) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/oslo_db/sqlalchemy/engines.py", line 152, in create_engine 2016-06-02 21:40:33.478 78979 ERROR cinder test_conn = _test_connection(engi ne, max_retries, retry_interval) 2016-06-02 21:40:33.478 78979 ERROR cinder File "/usr/lib/python2.7/site-packa ges/oslo_db/sqlalchemy/engines.py", line 330, in _test_connection 2016-06-02 21:40:33.478 78979 ERROR cinder time.sleep(retry_interval) 2016-06-02 21:40:33.478 78979 ERROR cinder KeyboardInterrupt 2016-06-02 21:40:33.478 78979 ERROR cinder 2016-06-03 23:18:14.146 144312 DEBUG oslo_db.api [req-59c2563b-905b-4b88-9fa9-a1 7bf1f4b447 - - - - -] Loading backend 'sqlalchemy' from 'cinder.db.sqlalchemy.ap i' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:230 2016-06-03 23:18:14.152 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 10 attempts left. 2016-06-03 23:18:24.163 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 9 attempts left. 2016-06-03 23:18:34.174 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 8 attempts left. 2016-06-03 23:18:44.185 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 7 attempts left. 2016-06-03 23:18:54.195 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 6 attempts left. 2016-06-03 23:19:04.206 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 5 attempts left. 2016-06-03 23:19:14.216 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 4 attempts left. 2016-06-03 23:19:24.227 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 3 attempts left. 2016-06-03 23:19:34.237 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b- 905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 2 attempts left. Cinder-volume Service 2016-06-03 23:13:53.049 144263 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198 2016-06-03 23:13:53.049 144263 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211 2016-06-03 23:13:53.303 144263 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'cinder.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:230 2016-06-03 23:13:53.592 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left. 2016-06-03 23:14:03.603 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left. 2016-06-03 23:14:13.614 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left. 2016-06-03 23:14:23.625 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 7 attempts left. 2016-06-03 23:14:33.636 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 6 attempts left. 2016-06-03 23:14:43.646 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 5 attempts left. 2016-06-03 23:14:53.657 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 4 attempts left. 2016-06-03 23:15:03.668 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 3 attempts left. 2016-06-03 23:15:13.679 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 2 attempts left. 2016-06-03 23:15:23.690 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 1 attempts left. 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume [-] Volume service pun01cloud03.ad.infosys.com@lvm failed to start. 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume Traceback (most recent call last): 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 76, in main 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume binary='cinder-volume') 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/service.py", line 251, in create 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume service_name=service_name) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/service.py", line 130, in __init__ 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume args, *kwargs) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 203, in __init__ 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume args, *kwargs) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 146, in __init__ 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume super(SchedulerDependentManager, self).__init__(host, db_driver) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 87, in __init__ 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume super(Manager, self).__init__(db_driver) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/db/base.py", line 42, in __init__ 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume self.db.dispose_engine() 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 84, in dispose_engine 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume if 'sqlite' not in IMPL.get_engine().name: 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 87, in get_engine 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume facade = _create_facade_lazily() 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 74, in _create_facade_lazily 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume *dict(CONF.database) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__ 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume slave_connection=slave_connection) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume engine_args, maker_args) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume sql_connection=sql_connection, *engine_kwargs) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume test_conn = _test_connection(engine, max_retries, retry_interval) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 334, in _test_connection 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume six.reraise(type(de_ref), de_ref) 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume File "<string>", line 2, in reraise 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.168.170.30' (113)") 2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume 2016-06-03 23:15:33.718 144263 ERROR cinder.cmd.volume [-] No volume service(s) started successfully, terminating.

Storage - Cinder - lvm driver Configuration [lvm]

volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_group = cinder-volumes iscsi_protocol = iscsi iscsi_helper = tgtadm

Cinder Volume service is down on Controller Node

Dear OpenStack Cinder Experts,

Need your help on solving one issue related to Cinder Service where following is the scenario. I am using three Node configuration for deploying Openstack Liberty release on RHEL OS. I am not able to start Cinder-volume Service on Storage node and also while debugging found that " it is not able to initialize Driver" and fails . given below is the Controleller and storage log. Please advise.

Cinder-Scheduler.log (Controller Node) Node)

2016-06-01 20:23:41.760 24572 CRITICAL cinder [req-2b9c0274-a846-4767-9865-b55b337cc60d - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError)
 (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.binary services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.rpc_available_version AS services_rpc_available_version, services.object_current_version AS services_object_current_version, services.object_available_version AS services_object_available_version \nFROM services \nWHERE services.deleted = false AND services.topic = %s AND services.disabled = false'] [parameters: ('cinder-volume',)]
2016-06-01 20:23:41.760 24572 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.binary services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.rpc_available_version AS services_rpc_available_version, services.object_current_version AS services_object_current_version, services.object_available_version AS services_object_available_version \nFROM services \nWHERE services.deleted = false AND services.topic = %s AND services.disabled = false'] [parameters: ('cinder-volume',)]

('cinder-volume',)]

Cinder-volume log ******log


 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2239
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager [req-357b18bf-274c-4a8e-a65e-54f8362cd542 - - - - -] Failed to initialize driver.
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Traceback (most recent call last):
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 368, in init_host
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     self.driver.check_for_setup_error()
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     return f(args, *kwargs)
f(*args, **kwargs)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 269, in check_for_setup_error
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     lvm_conf=lvm_conf_file)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 86, in __init__
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     if self._vg_exists() is False:
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 123, in _vg_exists
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     run_as_root=True)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 155, in execute
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     return processutils.execute(cmd, processutils.execute(*cmd, **kwargs)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager     cmd=sanitized_cmd)
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Command: env LC_ALL=C vgs --noheadings -o name cinder-volumes
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Exit code: 5
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Stdout: u''
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager Stderr: u'  Volume group "cinder-volumes" not found\n  Cannot process volume group cinder-volumes\n'
2016-06-03 23:12:02.876 3171 ERROR cinder.volume.manager
2016-06-03 23:12:02.895 3171 DEBUG oslo_db.sqlalchemy.engines [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Parent process 3152 forked (3171) with an open database connection, which is being discarded and recreated. checkout /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:357
2016-06-03 23:12:02.904 3171 DEBUG cinder.service [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Creating RPC server for service cinder-volume start /usr/lib/python2.7/site-packages/cinder/service.py:155
2016-06-03 23:12:02.926 3171 DEBUG oslo_messaging._drivers.amqp [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Pool creating new connection create /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:103
2016-06-03 23:12:02.933 3171 INFO oslo.messaging._drivers.impl_rabbit [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Connecting to AMQP server on 192.168.170.30:5672
2016-06-03 23:12:10.944 3171 INFO oslo.messaging._drivers.impl_rabbit [req-372ad387-80ff-4b60-90d7-85d0de94e83b - - - - -] Connected to AMQP server on 192.168.170.30:5672
2016-06-03 23:12:20.959 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down".
2016-06-03 23:12:30.963 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down".
2016-06-03 23:12:40.963 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down".
2016-06-03 23:12:50.973 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down".
2016-06-03 23:13:00.978 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down".
2016-06-03 23:13:10.978 3171 ERROR cinder.service [-] Manager for service cinder-volume pun01cloud01.ad.infosys.com is reporting problems, not sending heartbeat. Service will appear "down".
2016-06-03 23:13:11.452 3171 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting
2016-06-03 23:13:11.454 3152 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting

exiting

[root@pun01cloud01 cinder]# cinder service-list +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | pun01cloud01.ad.infosys.com | nova | enabled | up | 2016-06-06T11:24:48.000000 | - | | cinder-volume | pun01cloud01.ad.infosys.com | nova | enabled | down | - | - | +------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+

+------------------+-----------------------------+------+---------+-------+----------------------------+-----------------+

Storage Node (cinder-manage) (cinder-manage)

2016-06-02 21:40:00.253 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d                                                                                        8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 10 attempts left.
2016-06-02 21:40:10.264 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d                                                                                        8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 9 attempts left.
2016-06-02 21:40:20.275 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d                                                                                        8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 8 attempts left.
2016-06-02 21:40:30.285 78979 WARNING oslo_db.sqlalchemy.engines [req-2d407b86-d                                                                                        8ca-4b75-a932-0acf1e4e65f5 - - - - -] SQL connection failed. 7 attempts left.
2016-06-02 21:40:33.478 78979 CRITICAL cinder [req-2d407b86-d8ca-4b75-a932-0acf1                                                                                        e4e65f5 - - - - -] KeyboardInterrupt
2016-06-02 21:40:33.478 78979 ERROR cinder Traceback (most recent call last):
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/bin/cinder-manage", line                                                                                         10, in <module>
2016-06-02 21:40:33.478 78979 ERROR cinder     sys.exit(main())
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/cmd/manage.py", line 592, in main
2016-06-02 21:40:33.478 78979 ERROR cinder     fn(fn_args)
fn(*fn_args)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/cmd/manage.py", line 439, in list
2016-06-02 21:40:33.478 78979 ERROR cinder     services = objects.ServiceList.ge                                                                                        t_all(ctxt)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/oslo_versionedobjects/base.py", line 171, in wrapper
2016-06-02 21:40:33.478 78979 ERROR cinder     result = fn(cls, context, *args,                                                                                         *kwargs)
**kwargs)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/objects/service.py", line 119, in get_all
2016-06-02 21:40:33.478 78979 ERROR cinder     services = db.service_get_all(con                                                                                        text, filters)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/db/api.py", line 110, in service_get_all
2016-06-02 21:40:33.478 78979 ERROR cinder     return IMPL.service_get_all(conte                                                                                        xt, disabled)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/db/sqlalchemy/api.py", line 165, in wrapper
2016-06-02 21:40:33.478 78979 ERROR cinder     return f(args, *kwargs)
f(*args, **kwargs)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/db/sqlalchemy/api.py", line 365, in service_get_all
2016-06-02 21:40:33.478 78979 ERROR cinder     query = model_query(context, mode                                                                                        ls.Service)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/db/sqlalchemy/api.py", line 242, in model_query
2016-06-02 21:40:33.478 78979 ERROR cinder     session = kwargs.get('session') o                                                                                        r get_session()
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/db/sqlalchemy/api.py", line 92, in get_session
2016-06-02 21:40:33.478 78979 ERROR cinder     facade = _create_facade_lazily()
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/cinder/db/sqlalchemy/api.py", line 74, in _create_facade_lazily
2016-06-02 21:40:33.478 78979 ERROR cinder     *dict(CONF.database)
**dict(CONF.database)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__
2016-06-02 21:40:33.478 78979 ERROR cinder     slave_connection=slave_connection                                                                                        )
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
2016-06-02 21:40:33.478 78979 ERROR cinder     engine_args, maker_args)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection
2016-06-02 21:40:33.478 78979 ERROR cinder     sql_connection=sql_connection, * **                                                                                        engine_kwargs)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
2016-06-02 21:40:33.478 78979 ERROR cinder     test_conn = _test_connection(engi                                                                                        ne, max_retries, retry_interval)
2016-06-02 21:40:33.478 78979 ERROR cinder   File "/usr/lib/python2.7/site-packa                                                                                        ges/oslo_db/sqlalchemy/engines.py", line 330, in _test_connection
2016-06-02 21:40:33.478 78979 ERROR cinder     time.sleep(retry_interval)
2016-06-02 21:40:33.478 78979 ERROR cinder KeyboardInterrupt
2016-06-02 21:40:33.478 78979 ERROR cinder
2016-06-03 23:18:14.146 144312 DEBUG oslo_db.api [req-59c2563b-905b-4b88-9fa9-a1                                                                                        7bf1f4b447 - - - - -] Loading backend 'sqlalchemy' from 'cinder.db.sqlalchemy.ap                                                                                        i' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:230
2016-06-03 23:18:14.152 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 10 attempts left.
2016-06-03 23:18:24.163 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 9 attempts left.
2016-06-03 23:18:34.174 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 8 attempts left.
2016-06-03 23:18:44.185 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 7 attempts left.
2016-06-03 23:18:54.195 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 6 attempts left.
2016-06-03 23:19:04.206 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 5 attempts left.
2016-06-03 23:19:14.216 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 4 attempts left.
2016-06-03 23:19:24.227 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 3 attempts left.
2016-06-03 23:19:34.237 144312 WARNING oslo_db.sqlalchemy.engines [req-59c2563b-                                                                                        905b-4b88-9fa9-a17bf1f4b447 - - - - -] SQL connection failed. 2 attempts left.
 Cinder-volume Service
 2016-06-03 23:13:53.049 144263 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
2016-06-03 23:13:53.049 144263 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
2016-06-03 23:13:53.303 144263 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'cinder.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:230
2016-06-03 23:13:53.592 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.
2016-06-03 23:14:03.603 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.
2016-06-03 23:14:13.614 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.
2016-06-03 23:14:23.625 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 7 attempts left.
2016-06-03 23:14:33.636 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 6 attempts left.
2016-06-03 23:14:43.646 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 5 attempts left.
2016-06-03 23:14:53.657 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 4 attempts left.
2016-06-03 23:15:03.668 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 3 attempts left.
2016-06-03 23:15:13.679 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 2 attempts left.
2016-06-03 23:15:23.690 144263 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 1 attempts left.
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume [-] Volume service pun01cloud03.ad.infosys.com@lvm failed to start.
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume Traceback (most recent call last):
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 76, in main
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     binary='cinder-volume')
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 251, in create
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     service_name=service_name)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 130, in __init__
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     args, *kwargs)
*args, **kwargs)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 203, in __init__
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     args, *kwargs)
*args, **kwargs)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 146, in __init__
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     super(SchedulerDependentManager, self).__init__(host, db_driver)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 87, in __init__
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     super(Manager, self).__init__(db_driver)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/base.py", line 42, in __init__
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     self.db.dispose_engine()
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 84, in dispose_engine
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     if 'sqlite' not in IMPL.get_engine().name:
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 87, in get_engine
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     facade = _create_facade_lazily()
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 74, in _create_facade_lazily
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     *dict(CONF.database)
**dict(CONF.database)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     slave_connection=slave_connection)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     engine_args, maker_args)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     sql_connection=sql_connection, *engine_kwargs)
**engine_kwargs)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     test_conn = _test_connection(engine, max_retries, retry_interval)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 334, in _test_connection
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume     six.reraise(type(de_ref), de_ref)
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume   File "<string>", line 2, in reraise
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.168.170.30' (113)")
2016-06-03 23:15:33.700 144263 ERROR cinder.cmd.volume
2016-06-03 23:15:33.718 144263 ERROR cinder.cmd.volume [-] No volume service(s) started successfully, terminating.

terminating.

Storage - Cinder - lvm driver Configuration [lvm]

 [lvm]

volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
iscsi_helper = tgtadm 

tgtadm