Ask Your Question

Revision history [back]

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... problem is with the same cinder.conf settings i am now getting an error:

2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume [req-21bd8854-fb9f-4368-ad48-876496524c5c - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:

2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume [req-21bd8854-fb9f-4368-ad48-876496524c5c - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:

2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume [req-21bd8854-fb9f-4368-ad48-876496524c5c - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSD_NAS
nas_volume_prov_type = thin

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:

2017-02-28 18:53:59.266 30279 ERROR cinder.cmd.volume [req-21bd8854-fb9f-4368-ad48-876496524c5c 19:13:45.053 10056 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 19:13:45.068 10056 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 19:13:45.073 10056 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 19:13:45.247 10056 INFO cinder.volume.manager [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 19:13:45.253 10056 WARNING cinder.keymgr.conf_key_mgr [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 18:53:59.266 30279 19:13:45.254 10056 ERROR cinder.cmd.volume

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSD_NAS
nas_volume_prov_type = thin

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:

2017-02-28 19:13:45.053 10056 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 19:13:45.068 10056 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 19:13:45.073 10056 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 19:13:45.247 10056 INFO cinder.volume.manager [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 19:13:45.253 10056 WARNING cinder.keymgr.conf_key_mgr [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSD_NAS
SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:

debug starting the service.

2017-02-28 19:13:45.053 10056 [root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 19:13:45.068 10056 20:06:12.178 17241 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 19:13:45.073 10056 20:06:12.184 17241 INFO cinder.rpc [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 19:13:45.247 10056 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 19:13:45.253 10056 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume [req-6a30ad94-5f03-4f55-93ce-5a814603ee12 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 19:13:45.254 10056 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 19:13:45.254 10056 ERROR cinder.cmd.volume
20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

Update i have been through the other posts and still do not see a solution. I am thinking its a python library mismatch.... how would i check that? I

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

Update i have been through the other posts and still do not see a solution. I am thinking its a python library mismatch.... how would i check that? I

problem is with the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error:error: python modules exist

[root@platform2 site-packages]# ls -ald cinder*
drwxr-xr-x 24 root root 4096 Mar  2 09:07 cinder
drwxr-xr-x  2 root root  143 Mar  2 09:07 cinder-10.0.0-py2.7.egg-info
drwxr-xr-x  6 root root 4096 Feb 26 17:51 cinderclient
drwxr-xr-x  2 root root  136 Mar  2 19:09 cinder_data
drwxr-xr-x  2 root root  112 Mar  2 19:09 cinder_data-0.0.1-py2.7.egg-info
drwxr-xr-x  2 root root   44 Mar  2 09:07 cinder_tests.egg-info

[root@platform2 site-packages]# ls -ald gluster*
drwxr-xr-x 5 root root   96 Mar  1 10:38 gluster
drwxr-xr-x 2 root root  112 Mar  1 10:36 glustercli-0.2-py2.7.egg-info
drwxr-xr-x 4 root root   68 Mar  1 10:38 gluster_georep_tools
drwxr-xr-x 2 root root 4096 Mar  1 10:38 gluster_georep_tools-0.2-py2.7.egg-info
drwxr-xr-x 2 root root   90 Mar  2 19:07 gluster_stats
drwxr-xr-x 2 root root  147 Mar  2 19:07 gluster_stats-1.2.2.dist-info
[root@platform2 site-packages]#

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is with after the after upgrade to ocata same cinder.conf settings as i have had and i am now getting an error: python modules exist

[root@platform2 site-packages]# ls -ald cinder*
drwxr-xr-x 24 root root 4096 Mar  2 09:07 cinder
drwxr-xr-x  2 root root  143 Mar  2 09:07 cinder-10.0.0-py2.7.egg-info
drwxr-xr-x  6 root root 4096 Feb 26 17:51 cinderclient
drwxr-xr-x  2 root root  136 Mar  2 19:09 cinder_data
drwxr-xr-x  2 root root  112 Mar  2 19:09 cinder_data-0.0.1-py2.7.egg-info
drwxr-xr-x  2 root root   44 Mar  2 09:07 cinder_tests.egg-info

[root@platform2 site-packages]# ls -ald gluster*
drwxr-xr-x 5 root root   96 Mar  1 10:38 gluster
drwxr-xr-x 2 root root  112 Mar  1 10:36 glustercli-0.2-py2.7.egg-info
drwxr-xr-x 4 root root   68 Mar  1 10:38 gluster_georep_tools
drwxr-xr-x 2 root root 4096 Mar  1 10:38 gluster_georep_tools-0.2-py2.7.egg-info
drwxr-xr-x 2 root root   90 Mar  2 19:07 gluster_stats
drwxr-xr-x 2 root root  147 Mar  2 19:07 gluster_stats-1.2.2.dist-info
[root@platform2 site-packages]#

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - No module named glusterfs

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is after the upgrade to ocata same cinder.conf settings as i have had and i am now getting an error: python modules exist

[root@platform2 site-packages]# ls -ald cinder*
drwxr-xr-x 24 root root 4096 Mar  2 09:07 cinder
drwxr-xr-x  2 root root  143 Mar  2 09:07 cinder-10.0.0-py2.7.egg-info
drwxr-xr-x  6 root root 4096 Feb 26 17:51 cinderclient
drwxr-xr-x  2 root root  136 Mar  2 19:09 cinder_data
drwxr-xr-x  2 root root  112 Mar  2 19:09 cinder_data-0.0.1-py2.7.egg-info
drwxr-xr-x  2 root root   44 Mar  2 09:07 cinder_tests.egg-info

[root@platform2 site-packages]# ls -ald gluster*
drwxr-xr-x 5 root root   96 Mar  1 10:38 gluster
drwxr-xr-x 2 root root  112 Mar  1 10:36 glustercli-0.2-py2.7.egg-info
drwxr-xr-x 4 root root   68 Mar  1 10:38 gluster_georep_tools
drwxr-xr-x 2 root root 4096 Mar  1 10:38 gluster_georep_tools-0.2-py2.7.egg-info
drwxr-xr-x 2 root root   90 Mar  2 19:07 gluster_stats
drwxr-xr-x 2 root root  147 Mar  2 19:07 gluster_stats-1.2.2.dist-info
[root@platform2 site-packages]#

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - CentOS7 / RDO / GlusterFS3.8 No module named glusterfs

still no solution...

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is after the upgrade to ocata same cinder.conf settings as i have had and i am now getting an error: python modules exist

[root@platform2 site-packages]# ls -ald cinder*
drwxr-xr-x 24 root root 4096 Mar  2 09:07 cinder
drwxr-xr-x  2 root root  143 Mar  2 09:07 cinder-10.0.0-py2.7.egg-info
drwxr-xr-x  6 root root 4096 Feb 26 17:51 cinderclient
drwxr-xr-x  2 root root  136 Mar  2 19:09 cinder_data
drwxr-xr-x  2 root root  112 Mar  2 19:09 cinder_data-0.0.1-py2.7.egg-info
drwxr-xr-x  2 root root   44 Mar  2 09:07 cinder_tests.egg-info

[root@platform2 site-packages]# ls -ald gluster*
drwxr-xr-x 5 root root   96 Mar  1 10:38 gluster
drwxr-xr-x 2 root root  112 Mar  1 10:36 glustercli-0.2-py2.7.egg-info
drwxr-xr-x 4 root root   68 Mar  1 10:38 gluster_georep_tools
drwxr-xr-x 2 root root 4096 Mar  1 10:38 gluster_georep_tools-0.2-py2.7.egg-info
drwxr-xr-x 2 root root   90 Mar  2 19:07 gluster_stats
drwxr-xr-x 2 root root  147 Mar  2 19:07 gluster_stats-1.2.2.dist-info
[root@platform2 site-packages]#

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - CentOS7 / RDO / GlusterFS3.8 No module named glusterfs

still no solution...From the cinder release notes:

--- upgrade: - The GlusterFS volume driver, which was deprecated in the Newton release, has been removed.

Any idea how i can get this back?

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is after the upgrade to ocata same cinder.conf settings as i have had and i am now getting an error: python modules exist

[root@platform2 site-packages]# ls -ald cinder*
drwxr-xr-x 24 root root 4096 Mar  2 09:07 cinder
drwxr-xr-x  2 root root  143 Mar  2 09:07 cinder-10.0.0-py2.7.egg-info
drwxr-xr-x  6 root root 4096 Feb 26 17:51 cinderclient
drwxr-xr-x  2 root root  136 Mar  2 19:09 cinder_data
drwxr-xr-x  2 root root  112 Mar  2 19:09 cinder_data-0.0.1-py2.7.egg-info
drwxr-xr-x  2 root root   44 Mar  2 09:07 cinder_tests.egg-info

[root@platform2 site-packages]# ls -ald gluster*
drwxr-xr-x 5 root root   96 Mar  1 10:38 gluster
drwxr-xr-x 2 root root  112 Mar  1 10:36 glustercli-0.2-py2.7.egg-info
drwxr-xr-x 4 root root   68 Mar  1 10:38 gluster_georep_tools
drwxr-xr-x 2 root root 4096 Mar  1 10:38 gluster_georep_tools-0.2-py2.7.egg-info
drwxr-xr-x 2 root root   90 Mar  2 19:07 gluster_stats
drwxr-xr-x 2 root root  147 Mar  2 19:07 gluster_stats-1.2.2.dist-info
[root@platform2 site-packages]#

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.

ocata upgrade - CentOS7 / RDO / GlusterFS3.8 No module named glusterfs

From the cinder release notes:

--- upgrade: - The GlusterFS volume driver, which was deprecated in the Newton release, has been removed.

Any idea how i can get this back?

gluster volumes and all visiable and mountable... validated all this as i can see instances in glance and volumes... controller even has a scratch mount share fomr the same gluster (/openstack_scratch type fuse.glusterfs) so i know its working...

problem is after the upgrade to ocata same cinder.conf settings as i have had and i am now getting an error: python modules exist

[root@platform2 site-packages]# ls -ald cinder*
drwxr-xr-x 24 root root 4096 Mar  2 09:07 cinder
drwxr-xr-x  2 root root  143 Mar  2 09:07 cinder-10.0.0-py2.7.egg-info
drwxr-xr-x  6 root root 4096 Feb 26 17:51 cinderclient
drwxr-xr-x  2 root root  136 Mar  2 19:09 cinder_data
drwxr-xr-x  2 root root  112 Mar  2 19:09 cinder_data-0.0.1-py2.7.egg-info
drwxr-xr-x  2 root root   44 Mar  2 09:07 cinder_tests.egg-info

[root@platform2 site-packages]# ls -ald gluster*
drwxr-xr-x 5 root root   96 Mar  1 10:38 gluster
drwxr-xr-x 2 root root  112 Mar  1 10:36 glustercli-0.2-py2.7.egg-info
drwxr-xr-x 4 root root   68 Mar  1 10:38 gluster_georep_tools
drwxr-xr-x 2 root root 4096 Mar  1 10:38 gluster_georep_tools-0.2-py2.7.egg-info
drwxr-xr-x 2 root root   90 Mar  2 19:07 gluster_stats
drwxr-xr-x 2 root root  147 Mar  2 19:07 gluster_stats-1.2.2.dist-info
[root@platform2 site-packages]#

debug starting the service.

[root@platform2 sahara]# 2017-02-28 20:06:11.782 17241 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-02-28 20:06:11.782 17241 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-02-28 20:06:11.783 17241 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-02-28 20:06:11.802 17241 WARNING oslo_config.cfg [-] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
2017-02-28 20:06:12.011 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.164 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-volume objects version 1.21 as minimum service version.
2017-02-28 20:06:12.178 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler objects version 1.21 as minimum service version.
2017-02-28 20:06:12.184 17241 INFO cinder.rpc [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Automatically selected cinder-scheduler RPC version 3.5 as minimum service version.
2017-02-28 20:06:12.298 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.334 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.334 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.340 17241 WARNING cinder.keymgr.conf_key_mgr [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] This key manager is insecure and is not recommended for production deployments
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Volume service platform2.orbit8.com@SSHD_NAS failed to start.
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     cluster=cluster)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 207, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     *args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 249, in __init__
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     active_backend_id=curr_active_backend_id)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     return import_class(import_str)(*args, **kwargs)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume     __import__(mod_str)
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume ImportError: No module named glusterfs
2017-02-28 20:06:12.341 17241 ERROR cinder.cmd.volume 
2017-02-28 20:06:12.384 17241 DEBUG oslo_db.sqlalchemy.engines [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-02-28 20:06:12.414 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Determined volume DB was not empty at startup.
2017-02-28 20:06:12.414 17241 DEBUG cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Cinder Volume DB check: vol_db_empty=False __init__ /usr/lib/python2.7/site-packages/cinder/volume/manager.py:215
2017-02-28 20:06:12.429 17241 INFO cinder.volume.manager [req-d8c9e611-7236-46d0-ba93-d68cb911a093 - - - - -] Image-volume cache disabled for host platform2.orbit8.com@SSD_NAS.

Cinder.conf for the volumes:

[SSD_NAS]
volume_driver= cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config= /etc/cinder/nfs_shares.conf
volume_backend_name= SSD_NAS
nfs_sparsed_volumes = True
[SSHD_NAS]
volume_driver= cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config= /etc/cinder/glusterfs_shares.conf
volume_backend_name= SSHD_NAS
nas_volume_prov_type = thin

System is Ocata, gluster fs version is 3.8 centos 7.