Cinder not starting after upgrade from liberty to newton
Hello
I am trying to upgrade openstack form liberty to newton. All the services went smooth and everything worked except the cinder service. Cinder services on the controller node seem to be ok and they are started, but cinder service on the block storage node fails to start. from the logs i see the following:
2017-03-29 16:50:09.159 1871 ERROR cinder.cmd.volume [req-333d6725-2e9e-47c0-9760-81784d9d6a3a - - - - -] No volume service(s) started successfully, terminating.
2017-03-29 16:50:10.870 1916 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-03-29 16:50:10.947 1916 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
2017-03-29 16:50:10.980 1916 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume [req-fcf4cf55-8ea8-4c93-994c-a1f5161656ab - - - - -] Volume service block1@lvm failed to start.
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 100, in main
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume cluster=cluster)
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 382, in create
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume cluster=cluster)
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 150, in __init__
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume self.is_upgrading_to_n = self.is_svc_upgrading_to_n(binary)
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 223, in is_svc_upgrading_to_n
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume return rpc_api.determine_obj_version_cap() == '1.3'
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 233, in determine_obj_version_cap
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume cinder.context.get_admin_context(), cls.BINARY)
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 188, in get_minimum_obj_version
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume binary)
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 173, in _get_minimum_version
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume raise exception.ServiceTooOld(msg)
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume ServiceTooOld: One of the services is in Liberty version. We do not provide backward compatibility with Liberty now, you need to upgrade to Mitaka first.
2017-03-29 16:50:11.166 1916 ERROR cinder.cmd.volume
2017-03-29 16:50:11.168 1916 ERROR cinder.cmd.volume [req-fcf4cf55-8ea8-4c93-994c-a1f5161656ab - - - - -] No volume service(s ...
Probably have an orphaned cinder-volume service in the DB, check services db table. Thats why is saying the error ServiceTooOld. After upgrade all components sigup cinder services using rpc queue. Check this: https://docs.openstack.org/developer/...
Also ensure this is: https://docs.openstack.org/developer/...
Thank you very much. I followed your advice and i deleted the rows on the cinder database to the service table and then i synced the database again and everything worked like a charm. Again thank you very much.