cinder containers are restarting again again

asked 2019-05-15 06:35:36 -0500

niket gravatar image

updated 2019-05-16 03:14:39 -0500

Using openstack kolla as my infrastructure. trying to bring cinder service up but cinder containers (cinder_scheduler, cinder_volume, cinder_backup and iscsid) are restarting again n again.

#globals.yml
enable_cinder: "yes"
enable_cinder_backend_lvm: "yes"

Could you please guide me on the same.

logs for your reference

cinder-backup.log

2019-05-14 17:52:09.233 6 INFO cinder.cmd.backup [-] Backup running in single process mode.
2019-05-14 17:52:09.739 6 CRITICAL cinder [req-2eb0d613-3509-49f1-8927-fd6859437616 - - - - -] Unhandled error: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.cluster_name AS services_cluster_name, services.host AS services_host, services.`binary` AS services_binary, services.updated_at AS services_updated_at, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %(binary_1)s'] [parameters: {u'binary_1': 'cinder-backup'}] (Background on this error at: http://sqlalche.me/e/f405)
2019-05-14 17:52:09.739 6 ERROR cinder Traceback (most recent call last):
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/bin/cinder-backup", line 10, in <module>
2019-05-14 17:52:09.739 6 ERROR cinder     sys.exit(main())
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/cmd/backup.py", line 112, in main
2019-05-14 17:52:09.739 6 ERROR cinder     process_number=1)
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/service.py", line 392, in create
2019-05-14 17:52:09.739 6 ERROR cinder     cluster=cluster, **kwargs)
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/service.py", line 155, in __init__
2019-05-14 17:52:09.739 6 ERROR cinder     *args, **kwargs)
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 117, in __init__
2019-05-14 17:52:09.739 6 ERROR cinder     self.backup_rpcapi = backup_rpcapi.BackupAPI()
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/rpc.py", line 207, in __init__
2019-05-14 17:52:09.739 6 ERROR cinder     obj_version_cap = self.determine_obj_version_cap()
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/rpc.py", line 260, in determine_obj_version_cap
2019-05-14 17:52:09.739 6 ERROR cinder     cinder.context.get_admin_context(), cls.BINARY)
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/objects/service.py", line 222, in get_minimum_obj_version
2019-05-14 17:52:09.739 6 ERROR cinder     binary)
2019-05-14 17:52:09.739 6 ERROR cinder   File "/var/lib ...
(more)
edit retag flag offensive close merge delete

Comments

The corresponding logs should tell you why the containers stop. However I don't know where logs are kept in a Kolla deployment.

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-15 06:55:50 -0500 )edit

Hi Bernd,

Thanks for your quick response. I have gone through logs also. it didn't help out.

unable to attach logs here.

niket gravatar imageniket ( 2019-05-16 03:04:36 -0500 )edit

I took the liberty to move the logs to your question and format them.

Cinder can't start because of this error: Table 'cinder.services' doesn't exist. For some reason, the Cinder database tables were not created. Does Kolla have an installation log?

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-16 03:18:13 -0500 )edit

All kola logs you can find at /var/lib/docker/volumes/kolla_logs/_data/cinder other then this , i don't thing there are any logs which kolla maintains.

niket gravatar imageniket ( 2019-05-16 04:49:14 -0500 )edit

I mean log messages generated when Kolla sets up OpenStack. As part of the process, it should create the Cinder database. This step was either not performed or it failed. The Kolla deployment logs, if there are any, should help you understand what prevented the DB creation.

Perhaps the Ansible log?

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-16 07:06:13 -0500 )edit