cinder-volume service not able to talk to cinder-scheduler , new volume creation failing

asked 2018-11-17 12:00:54 -0500

anonymous user

Anonymous

I have recently started seeing issues on the storage node where I have rbd driver configured as well as this node also act as a compute node for vms. This node was working perfectly fine for sometime back but I just did some cleaning on my ceph storage cluster by removing the bad osd's out of the cluster and post that I restarted some of my cinder services on controller and storage node and I started to see all issues on this node with below error. I can confirm that my controller node is absolutely fine as I have other zones using cinder-volume service and they are working fine using the same controller.

I also tried below options already. 1. restarted storage node physically as well as all relevant services. 2. I also checked in cinder service-list and the volume service is enabled and up in the controller. 3. I also tried direct command line and cinder-api accepts the request and creates the volume but it goes from creating --> error state and just ends there. 4. Regularly checking the cinder-scheduler service to identify cinder-volume host update but it's not able to see the host. 5. I also tried to manually create a volume in ceph on the same node using rbd driver commands and it works so connectivity from cinder-volume to ceph is also good.

I don't know what has gone wrong all of a sudden , below are the logs I am seeing, I would sincerely appreciate any help.

Openstack Version: Kilo ceph storage cluster: ceph cluster Glance is working too.

2018-11-17 17:45:40.444 32054 INFO cinder.volume.manager [req-d4d12536-3d95-423e-869f-836178cfcc83 - - - - -] Updating volume status 2018-11-17 17:45:40.445 32054 WARNING cinder.volume.manager [req-d4d12536-3d95-423e-869f-836178cfcc83 - - - - -] Unable to update stats, RBDDriver -1.1.0 (config name ceph06) driver is uninitialized. 2018-11-17 17:45:45.349 32044 DEBUG cinder.openstack.common.periodic_task [req-065127a6-a147-4737-9ffc-9a86a69da51a - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py:219 2018-11-17 17:45:45.350 32044 DEBUG cinder.openstack.common.periodic_task [req-065127a6-a147-4737-9ffc-9a86a69da51a - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py:219 2018-11-17 17:45:45.351 32044 INFO cinder.volume.manager [req-065127a6-a147-4737-9ffc-9a86a69da51a - - - - -] Updating volume status 2018-11-17 17:45:45.352 32044 WARNING cinder.volume.manager [req-065127a6-a147-4737-9ffc-9a86a69da51a - - - - -] Unable to update stats, RBDDriver -1.1.0 (config name ceph04) driver is uninitialized. 2018-11-17 17:46:01.622 32058 DEBUG cinder.openstack.common.periodic_task [req-b2cc861b-0524-4f27-982e-069ab9095b3a - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py:219 2018-11-17 17:46:01.623 32058 DEBUG cinder.openstack.common.periodic_task [req-b2cc861b-0524-4f27-982e-069ab9095b3a - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py:219 2018-11-17 17:46:01.624 32058 INFO cinder.volume.manager [req-b2cc861b-0524-4f27-982e-069ab9095b3a - - - - -] Updating volume status 2018-11-17 17:46:01.625 32058 WARNING cinder.volume.manager [req-b2cc861b-0524-4f27-982e-069ab9095b3a - - - - -] Unable to update stats, LVMISCSIDriver -3.0.0 (config name ceph) driver is uninitialized. 2018-11-17 17:46 ...

(more)
edit retag flag offensive close merge delete

Comments

Can you add log output from cinder-scheduler? Is the service up and running? If not, search the syslog and dmesg for hints.

eblock gravatar imageeblock ( 2018-11-19 02:20:37 -0500 )edit