Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Cinder volume keep going down after few seconds.

Hi we are first time installed devstack on RHEL 7 and We observe cinder-volume and cinder-backup going down after few seconds. Kindly help us we are stuck.

Please find below the logs of cinder-volume,cinder-backup and cinder-scheduler logs.

cinder-volume

(cinder-volume)[cinder@compute1 /]$ vi /var/log/kolla/cinder/cinder-volume.log 2019-02-06 11:43:20.568 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:43:30.568 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:43:31.402 33 WARNING cinder.volume.manager [req-ddc20724-8ced-4b4b-b8eb-09ab65fb59f4 - - - - -] Update driver status failed: (config name lvm-1) is uninitialized. 2019-02-06 11:43:40.569 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:43:50.569 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:44:00.570 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:44:10.571 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:44:20.571 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:44:30.572 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:44:31.403 33 WARNING cinder.volume.manager [req-ddc20724-8ced-4b4b-b8eb-09ab65fb59f4 - - - - -] Update driver status failed: (config name lvm-1) is uninitialized. 2019-02-06 11:44:40.573 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:44:50.573 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:45:00.574 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:45:10.574 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:45:20.575 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:45:30.576 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down". 2019-02-06 11:45:31.404 33 WARNING cinder.volume.manager [req-ddc20724-8ced-4b4b-b8eb-09ab65fb59f4 - - - - -] Update driver status failed: (config name lvm-1) is uninitialized. 2019-02-06 11:45:40.576 33 ERROR cinder.service [-] Manager for service cinder-volume compute1@lvm-1 is reporting problems, not sending heartbeat. Service will appear "down".

cinder-backup

(cinder-volume)[cinder@compute1 /]$ vi /var/log/kolla/cinder/cinder-backup.log 2019-02-06 11:50:50.426 6 ERROR os_brick.remotefs.remotefs [-] Failed to mount , reason: mount.nfs: remote share not in 'host:dir' format : ProcessExecutionError: Unexpected error while running command. 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'cinder.backup.manager.BackupManager._setup_backup_driver' failed: BrickException: NFS mount failed for share . Error - {'pnfs': u'Unexpected error while running command.\nCommand: mount -t nfs -o vers=4,minorversion=1 /var/lib/cinder/backup/d41d8cd98f00b204e9800998ecf8427e\nExit code: 32\nStdout: u\'\'\nStderr: u"mount.nfs: remote share not in \'host:dir\' format\n"', 'nfs': u'Unexpected error while running command.\nCommand: mount -t nfs -o /var/lib/cinder/backup/d41d8cd98f00b204e9800998ecf8427e\nExit code: 32\nStdout: u\'\'\nStderr: u"mount.nfs: remote share not in \'host:dir\' format\n"'} 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall Traceback (most recent call last): 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 187, in _run_loop 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall result = func(self.args, *self.kw) 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 172, in _setup_backup_driver 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall backup_service = self.get_backup_driver(ctxt) 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 144, in get_backup_driver 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall driver = driver_class(context=context, db=self.db) 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/nfs.py", line 61, in __init__ 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall backup_path = self._init_backup_repo_path() 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/nfs.py", line 81, in _init_backup_repo_path 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall remotefsclient.mount(self.backup_share) 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/remotefs/remotefs.py", line 105, in mount 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall self._mount_nfs(share, mount_path, flags) 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/remotefs/remotefs.py", line 151, in _mount_nfs 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall 'error': mnt_errors}) 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall BrickException: NFS mount failed for share . Error - {'pnfs': u'Unexpected error while running command.\nCommand: mount -t nfs -o vers=4,minorversion=1 /var/lib/cinder/backup/d41d8cd98f00b204e9800998ecf8427e\nExit code: 32\nStdout: u\'\'\nStderr: u"mount.nfs: remote share not in \'host:dir\' format\n"', 'nfs': u'Unexpected error while running command.\nCommand: mount -t nfs -o /var/lib/cinder/backup/d41d8cd98f00b204e9800998ecf8427e\nExit code: 32\nStdout: u\'\'\nStderr: u"mount.nfs: remote share not in \'host:dir\' format\n"'} 2019-02-06 11:50:50.428 6 ERROR oslo.service.loopingcall 2019-02-06 11:50:57.162 6 ERROR cinder.service [-] Manager for service cinder-backup compute1 is reporting problems, not sending heartbeat. Service will appear "down".

cinder-scheduler

[root@controller1 ~]# docker exec -it cinder_scheduler bash (cinder-scheduler)[cinder@controller1 /]$ vi /var/log/kolla/cinder/cinder-scheduler.log 2019-01-23 14:39:51.772 6 ERROR cinder.scheduler.flows.create_volume [req-1c9c1c3a-554b-4613-a991-8b3564602547 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available 2019-01-23 14:49:16.786 6 WARNING cinder.scheduler.host_manager [req-a55327d2-1ce3-4aa7-8344-b65e48dff098 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] volume service is down. (host: compute1@lvm-1) 2019-01-23 14:49:16.787 6 INFO cinder.scheduler.base_filter [req-a55327d2-1ce3-4aa7-8344-b65e48dff098 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Filtering removed all hosts for the request with volume ID '607cd57b-3f16-44e0-9823-6efdafd36a7a'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0) 2019-01-23 14:49:16.787 6 WARNING cinder.scheduler.filter_scheduler [req-a55327d2-1ce3-4aa7-8344-b65e48dff098 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] No weighed backend found for volume with properties: None 2019-01-23 14:49:16.788 6 INFO cinder.message.api [req-a55327d2-1ce3-4aa7-8344-b65e48dff098 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Creating message record for request_id = req-a55327d2-1ce3-4aa7-8344-b65e48dff098 2019-01-23 14:49:16.838 6 ERROR cinder.scheduler.flows.create_volume [req-a55327d2-1ce3-4aa7-8344-b65e48dff098 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available 2019-01-23 14:50:26.246 6 WARNING cinder.scheduler.host_manager [req-a75ee8fe-8131-44d9-a96a-548034c4f947 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] volume service is down. (host: compute1@lvm-1) 2019-01-23 14:50:26.247 6 INFO cinder.scheduler.base_filter [req-a75ee8fe-8131-44d9-a96a-548034c4f947 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Filtering removed all hosts for the request with volume ID '99ba3b39-65ed-4ef0-88a7-d2a072f02dc1'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0) 2019-01-23 14:50:26.248 6 WARNING cinder.scheduler.filter_scheduler [req-a75ee8fe-8131-44d9-a96a-548034c4f947 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] No weighed backend found for volume with properties: None 2019-01-23 14:50:26.249 6 INFO cinder.message.api [req-a75ee8fe-8131-44d9-a96a-548034c4f947 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Creating message record for request_id = req-a75ee8fe-8131-44d9-a96a-548034c4f947 2019-01-23 14:50:26.275 6 ERROR cinder.scheduler.flows.create_volume [req-a75ee8fe-8131-44d9-a96a-548034c4f947 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available 2019-01-23 14:56:49.974 6 WARNING cinder.scheduler.host_manager [req-c1ed2ad4-ef96-42f5-acf2-8c15810bc9b3 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] volume service is down. (host: compute1@lvm-1) 2019-01-23 14:56:49.975 6 INFO cinder.scheduler.base_filter [req-c1ed2ad4-ef96-42f5-acf2-8c15810bc9b3 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Filtering removed all hosts for the request with volume ID '0a00b846-ff7e-4341-9101-a1af2670743a'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0) 2019-01-23 14:56:49.977 6 WARNING cinder.scheduler.filter_scheduler [req-c1ed2ad4-ef96-42f5-acf2-8c15810bc9b3 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] No weighed backend found for volume with properties: None 2019-01-23 14:56:49.978 6 INFO cinder.message.api [req-c1ed2ad4-ef96-42f5-acf2-8c15810bc9b3 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Creating message record for request_id = req-c1ed2ad4-ef96-42f5-acf2-8c15810bc9b3 2019-01-23 14:56:50.040 6 ERROR cinder.scheduler.flows.create_volume [req-c1ed2ad4-ef96-42f5-acf2-8c15810bc9b3 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available 2019-01-23 15:01:16.799 6 WARNING cinder.scheduler.host_manager [req-c7960973-1187-4c9e-9e2e-7e8df87f3ab3 4d6457bc578043a1b34b70ea1d0ce559 1b315a2c067e469cb38329849a87f217 - default default] volume service is down. (host: compute1@lvm-1)