failed upgrade swift from 1.7 to 1.13 on centos 6.5

asked 2015-02-17 13:14:33 -0500

Jonathan gravatar image

updated 2015-02-18 14:52:05 -0500

We currently have swift 1.7 installed and running on CentOS 6.5 with the following .rpm packages and their dependencies:

openstack-swift-1.7.4-3.el6.noarch.rpm
openstack-swift-account-1.7.4-3.el6.noarch.rpm
openstack-swift-container-1.7.4-3.el6.noarch.rpm
openstack-swift-object-1.7.4-3.el6.noarch.rpm

We are trying to upgrade to swift 1.13 using the following .rpm packages and their dependencies:

openstack-swift-1.13.1-1.el6.noarch.rpm
openstack-swift-account-1.13.1-1.el6.noarch.rpm
openstack-swift-container-1.13.1-1.el6.noarch.rpm
openstack-swift-object-1.13.1-1.el6.noarch.rpm

I upgraded one storage node and then checked the object server log on the storage node. I see errors like the following:

Feb 18 20:24:12 localhost object-auditor: Quarantined object /srv/node/sdb1/objects/94795/607/5c92dc59a2056777ed13080fa1c35607/1424214855.39893.data: Hash of name in metadata does not match directory name
Feb 18 20:24:12 localhost object-auditor: ERROR Object /srv/node/sdb1/objects/94795/607/5c92dc59a2056777ed13080fa1c35607 failed audit and was quarantined: Hash of name in metadata does not match directory name

If I look at the files in the quarantined/object directories I see files uploaded earlier. I've checked the swift_hash_path_prefix and swift_hash_path_suffix in /etc/swift.conf and they are the same as before the upgrade. I am using the same ring files from before the upgrade.

Am I missing something or is it not possible to upgrade swift from 1.7 to 1.13?

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2015-02-18 15:48:03 -0500

updated 2015-02-18 15:52:11 -0500

There were potential issues when upgrading from version 1.6.0 or earlier to a version greater than 1.6.0, but that shouldn't be relevant to you. The workaround was either upgrade Swift on your ring building server last after all Swift nodes have been successfully upgraded, or refrain from generating rings until all Swift nodes have been successfully upgraded.

In your case... did you just upgrade the one storage node so far? If so, upgrading all the nodes followed by the ring generating server last is probably good advice. 1.7.x to 1.13.x is a big jump version-wise, but it's not that big!

edit flag offensive delete link more

Comments

I tried your suggestion of upgrading all the nodes followed by the ring gen server. However all that happened was that on start up, each storage node ran the object-auditor which promptly quarantined all objects as above. Its almost as if the behaviour of the hash_prefix/suffix has changed.

Jonathan gravatar imageJonathan ( 2015-02-18 16:32:24 -0500 )edit
1

answered 2015-02-18 18:17:12 -0500

Jonathan gravatar image

updated 2015-02-18 18:21:08 -0500

Problem solved. After surmising that the problem must be something to do with the hash prefix/suffix, I found bug 1157454 on launchpad, which shows that the hash prefix was added in swift 1.8. When our 1.7 deployment was setup, they must have followed the documentation for 1.8 or later and set both a hash prefix and suffix. Now 1.7 ignored the hash prefix but once 1.13 upgrade was done, the prefix caused the object-auditor to quarantine everything.

I have set the hash prefix to empty and now the upgrade is successful.

edit flag offensive delete link more

Comments

I was against that stupid prefix thing when it was introduced, but I wasn't on core back then and could not veto it.

zaitcev gravatar imagezaitcev ( 2015-02-24 20:12:32 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-02-17 13:09:19 -0500

Seen: 230 times

Last updated: Feb 18 '15