Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

service cinder-volumes fails to start

service cinder-volume restart fails

on controller node cinder service-list lists only scheduler service and volume service

root@controller:/home/stack/scripts/controller/v0.2# cinder service-list +------------------+------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | controller | nova | enabled | up | 2016-03-11T16:47:09.000000 | None | +------------------+------------+------+---------+-------+----------------------------+-----------------+

on the blkstorage node (10.0.0.41)

cinder-volume.log has following traces

2016-03-11 17:31:31.696 27695 TRACE cinder Traceback (most recent call last): 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/bin/cinder-volume", line 10, in <module> 2016-03-11 17:31:31.696 27695 TRACE cinder sys.exit(main()) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 73, in main 2016-03-11 17:31:31.696 27695 TRACE cinder binary='cinder-volume') 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 249, in create 2016-03-11 17:31:31.696 27695 TRACE cinder service_name=service_name) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 129, in __init__ 2016-03-11 17:31:31.696 27695 TRACE cinder args, *kwargs) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 211, in __init__ 2016-03-11 17:31:31.696 27695 TRACE cinder context.get_admin_context()) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 284, in _set_voldb_empty_at_startup_indicator 2016-03-11 17:31:31.696 27695 TRACE cinder vol_entries = self.db.volume_get_all(ctxt, None, 1, filters=None) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 208, in volume_get_all 2016-03-11 17:31:31.696 27695 TRACE cinder sort_dirs=sort_dirs, filters=filters) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 163, in wrapper 2016-03-11 17:31:31.696 27695 TRACE cinder return f(args, *kwargs) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 1353, in volume_get_all 2016-03-11 17:31:31.696 27695 TRACE cinder return query.all() 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2320, in all 2016-03-11 17:31:31.696 27695 TRACE cinder return list(self) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__ 2016-03-11 17:31:31.696 27695 TRACE cinder return self._execute_and_instances(context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances 2016-03-11 17:31:31.696 27695 TRACE cinder result = conn.execute(querycontext.statement, self._params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute 2016-03-11 17:31:31.696 27695 TRACE cinder return meth(self, multiparams, params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection 2016-03-11 17:31:31.696 27695 TRACE cinder return connection._execute_clauseelement(self, multiparams, params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement 2016-03-11 17:31:31.696 27695 TRACE cinder compiled_sql, distilled_params 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context 2016-03-11 17:31:31.696 27695 TRACE cinder context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception 2016-03-11 17:31:31.696 27695 TRACE cinder e, statement, parameters, cursor, context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception 2016-03-11 17:31:31.696 27695 TRACE cinder util.raise_from_cause(newraise, exc_info) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-03-11 17:31:31.696 27695 TRACE cinder reraise(type(exception), exception, tb=exc_tb) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context 2016-03-11 17:31:31.696 27695 TRACE cinder context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute 2016-03-11 17:31:31.696 27695 TRACE cinder cursor.execute(statement, parameters) 2016-03-11 17:31:31.696 27695 TRACE cinder OperationalError: (OperationalError) no such table: consistencygroups u'SELECT anon_1.volumes_created_at AS anon_1_volumes_created_at, anon_1.volumes_updated_at AS anon_1_volumes_updated_at, anon_1.volumes_deleted_at AS anon_1_volumes_deleted_at, anon_1.volumes_id AS anon_1_volumes_id, anon_1.volumes__name_id AS anon_1_volumes__name_id, anon_1.volumes_ec2_id AS anon_1_volumes_ec2_id, anon_1.volumes_user_id AS anon_1_volumes_user_id, anon_1.volumes_project_id AS anon_1_volumes_project_id, anon_1.volumes_snapshot_id AS anon_1_volumes_snapshot_id, anon_1.volumes_host AS anon_1_volumes_host, anon_1.volumes_size AS anon_1_volumes_size, anon_1.volumes_availability_zone AS anon_1_volumes_availability_zone, anon_1.volumes_status AS anon_1_volumes_status, anon_1.volumes_attach_status AS anon_1_volumes_attach_status, anon_1.volumes_migration_status AS anon_1_volumes_migration_status, anon_1.volumes_scheduled_at AS anon_1_volumes_scheduled_at, anon_1.volumes_launched_at AS anon_1_volumes_launched_at, anon_1.volumes_terminated_at AS anon_1_volumes_terminated_at, anon_1.volumes_display_name AS anon_1_volumes_display_name, anon_1.volumes_display_description AS anon_1_volumes_display_description, anon_1.volumes_provider_location AS anon_1_volumes_provider_location, anon_1.v

service cinder-volumes fails to start

service cinder-volume restart fails

on controller node cinder service-list lists only scheduler service and volume service

root@controller:/home/stack/scripts/controller/v0.2# cinder service-list +------------------+------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | controller | nova | enabled | up | 2016-03-11T16:47:09.000000 | None | +------------------+------------+------+---------+-------+----------------------------+-----------------+

on the blkstorage node (10.0.0.41)

cinder-volume.log has following traces

2016-03-11 17:31:31.696 27695 TRACE cinder Traceback (most recent call last): 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/bin/cinder-volume", line 10, in <module> 2016-03-11 17:31:31.696 27695 TRACE cinder sys.exit(main()) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 73, in main 2016-03-11 17:31:31.696 27695 TRACE cinder binary='cinder-volume') 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 249, in create 2016-03-11 17:31:31.696 27695 TRACE cinder service_name=service_name) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 129, in __init__ 2016-03-11 17:31:31.696 27695 TRACE cinder args, *kwargs) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 211, in __init__ 2016-03-11 17:31:31.696 27695 TRACE cinder context.get_admin_context()) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 284, in _set_voldb_empty_at_startup_indicator 2016-03-11 17:31:31.696 27695 TRACE cinder vol_entries = self.db.volume_get_all(ctxt, None, 1, filters=None) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 208, in volume_get_all 2016-03-11 17:31:31.696 27695 TRACE cinder sort_dirs=sort_dirs, filters=filters) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 163, in wrapper 2016-03-11 17:31:31.696 27695 TRACE cinder return f(args, *kwargs) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 1353, in volume_get_all 2016-03-11 17:31:31.696 27695 TRACE cinder return query.all() 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2320, in all 2016-03-11 17:31:31.696 27695 TRACE cinder return list(self) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__ 2016-03-11 17:31:31.696 27695 TRACE cinder return self._execute_and_instances(context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances 2016-03-11 17:31:31.696 27695 TRACE cinder result = conn.execute(querycontext.statement, self._params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute 2016-03-11 17:31:31.696 27695 TRACE cinder return meth(self, multiparams, params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection 2016-03-11 17:31:31.696 27695 TRACE cinder return connection._execute_clauseelement(self, multiparams, params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement 2016-03-11 17:31:31.696 27695 TRACE cinder compiled_sql, distilled_params 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context 2016-03-11 17:31:31.696 27695 TRACE cinder context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception 2016-03-11 17:31:31.696 27695 TRACE cinder e, statement, parameters, cursor, context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception 2016-03-11 17:31:31.696 27695 TRACE cinder util.raise_from_cause(newraise, exc_info) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-03-11 17:31:31.696 27695 TRACE cinder reraise(type(exception), exception, tb=exc_tb) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context 2016-03-11 17:31:31.696 27695 TRACE cinder context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute 2016-03-11 17:31:31.696 27695 TRACE cinder cursor.execute(statement, parameters) 2016-03-11 17:31:31.696 27695 TRACE cinder OperationalError: (OperationalError) no such table: consistencygroups u'SELECT anon_1.volumes_created_at AS anon_1_volumes_created_at, anon_1.volumes_updated_at AS anon_1_volumes_updated_at, anon_1.volumes_deleted_at AS anon_1_volumes_deleted_at, anon_1.volumes_id AS anon_1_volumes_id, anon_1.volumes__name_id AS anon_1_volumes__name_id, anon_1.volumes_ec2_id AS anon_1_volumes_ec2_id, anon_1.volumes_user_id AS anon_1_volumes_user_id, anon_1.volumes_project_id AS anon_1_volumes_project_id, anon_1.volumes_snapshot_id AS anon_1_volumes_snapshot_id, anon_1.volumes_host AS anon_1_volumes_host, anon_1.volumes_size AS anon_1_volumes_size, anon_1.volumes_availability_zone AS anon_1_volumes_availability_zone, anon_1.volumes_status AS anon_1_volumes_status, anon_1.volumes_attach_status AS anon_1_volumes_attach_status, anon_1.volumes_migration_status AS anon_1_volumes_migration_status, anon_1.volumes_scheduled_at AS anon_1_volumes_scheduled_at, anon_1.volumes_launched_at AS anon_1_volumes_launched_at, anon_1.volumes_terminated_at AS anon_1_volumes_terminated_at, anon_1.volumes_display_name AS anon_1_volumes_display_name, anon_1.volumes_display_description AS anon_1_volumes_display_description, anon_1.volumes_provider_location AS anon_1_volumes_provider_location, anon_1.v

service cinder-volumes fails to start

service cinder-volume restart fails

on controller node cinder service-list lists only scheduler service and volume service

root@controller:/home/stack/scripts/controller/v0.2# cinder service-list
+------------------+------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |    Host    | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | controller | nova | enabled |   up  | 2016-03-11T16:47:09.000000 |       None      |
+------------------+------------+------+---------+-------+----------------------------+-----------------+

+------------------+------------+------+---------+-------+----------------------------+-----------------+ on the blkstorage node (10.0.0.41)

cinder-volume.log has following traces

traces 2016-03-11 17:31:31.696 27695 TRACE cinder Traceback (most recent call last): 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/bin/cinder-volume", line 10, in <module> 2016-03-11 17:31:31.696 27695 TRACE cinder sys.exit(main()) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 73, in main 2016-03-11 17:31:31.696 27695 TRACE cinder binary='cinder-volume') 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 249, in create 2016-03-11 17:31:31.696 27695 TRACE cinder service_name=service_name) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 129, in __init__ 2016-03-11 17:31:31.696 27695 TRACE cinder args, *kwargs) *args, **kwargs) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 211, in __init__ 2016-03-11 17:31:31.696 27695 TRACE cinder context.get_admin_context()) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 284, in _set_voldb_empty_at_startup_indicator 2016-03-11 17:31:31.696 27695 TRACE cinder vol_entries = self.db.volume_get_all(ctxt, None, 1, filters=None) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 208, in volume_get_all 2016-03-11 17:31:31.696 27695 TRACE cinder sort_dirs=sort_dirs, filters=filters) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 163, in wrapper 2016-03-11 17:31:31.696 27695 TRACE cinder return f(args, *kwargs) f(*args, **kwargs) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 1353, in volume_get_all 2016-03-11 17:31:31.696 27695 TRACE cinder return query.all() 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2320, in all 2016-03-11 17:31:31.696 27695 TRACE cinder return list(self) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__ 2016-03-11 17:31:31.696 27695 TRACE cinder return self._execute_and_instances(context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances 2016-03-11 17:31:31.696 27695 TRACE cinder result = conn.execute(querycontext.statement, self._params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute 2016-03-11 17:31:31.696 27695 TRACE cinder return meth(self, multiparams, params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection 2016-03-11 17:31:31.696 27695 TRACE cinder return connection._execute_clauseelement(self, multiparams, params) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement 2016-03-11 17:31:31.696 27695 TRACE cinder compiled_sql, distilled_params 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context 2016-03-11 17:31:31.696 27695 TRACE cinder context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception 2016-03-11 17:31:31.696 27695 TRACE cinder e, statement, parameters, cursor, context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception 2016-03-11 17:31:31.696 27695 TRACE cinder util.raise_from_cause(newraise, exc_info) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-03-11 17:31:31.696 27695 TRACE cinder reraise(type(exception), exception, tb=exc_tb) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context 2016-03-11 17:31:31.696 27695 TRACE cinder context) 2016-03-11 17:31:31.696 27695 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute 2016-03-11 17:31:31.696 27695 TRACE cinder cursor.execute(statement, parameters) 2016-03-11 17:31:31.696 27695 TRACE cinder OperationalError: (OperationalError) no such table: consistencygroups u'SELECT anon_1.volumes_created_at AS anon_1_volumes_created_at, anon_1.volumes_updated_at AS anon_1_volumes_updated_at, anon_1.volumes_deleted_at AS anon_1_volumes_deleted_at, anon_1.volumes_id AS anon_1_volumes_id, anon_1.volumes__name_id AS anon_1_volumes__name_id, anon_1.volumes_ec2_id AS anon_1_volumes_ec2_id, anon_1.volumes_user_id AS anon_1_volumes_user_id, anon_1.volumes_project_id AS anon_1_volumes_project_id, anon_1.volumes_snapshot_id AS anon_1_volumes_snapshot_id, anon_1.volumes_host AS anon_1_volumes_host, anon_1.volumes_size AS anon_1_volumes_size, anon_1.volumes_availability_zone AS anon_1_volumes_availability_zone, anon_1.volumes_status AS anon_1_volumes_status, anon_1.volumes_attach_status AS anon_1_volumes_attach_status, anon_1.volumes_migration_status AS anon_1_volumes_migration_status, anon_1.volumes_scheduled_at AS anon_1_volumes_scheduled_at, anon_1.volumes_launched_at AS anon_1_volumes_launched_at, anon_1.volumes_terminated_at AS anon_1_volumes_terminated_at, anon_1.volumes_display_name AS anon_1_volumes_display_name, anon_1.volumes_display_description AS anon_1_volumes_display_description, anon_1.volumes_provider_location AS anon_1_volumes_provider_location, anon_1.v

anon_1.v