What causes swift ring rebalance to change location of container and object data on the disk?

asked 2014-01-23 01:31:13 -0500

fga gravatar image

Hi all,

We are using swift essex version in our development environment. Initially i had a swift cluster with 6 vm nodes with replication factor of 3 in a development environment. Due to disk error on the disk on the server hosting the vms 2 of the nodes went down. I tried to rebalance the rings by removing two corresponding nodes. In some cases request for container returns HTTP 404 meaning container does not exist eventhough request for account list returns that missing container. Then i issued following commands:

sudo swift-get-nodes /etc/swift/account.ring.gz AUTH_2

ssh "ls -lah /srv/node/sdb1/accounts/109606/a6e/6b0984ce96a01d328d6a9219401d0a6e/" ssh "ls -lah /srv/node/sdb1/accounts/109606/a6e/6b0984ce96a01d328d6a9219401d0a6e/" ssh "ls -lah /srv/node/sdb1/accounts/109606/a6e/6b0984ce96a01d328d6a9219401d0a6e/" ssh "ls -lah /srv/node/sdb1/accounts/109606/a6e/6b0984ce96a01d328d6a9219401d0a6e/"

As expected when i issue the command 3 of the servers as the .db file under the corresponding path on the disk.

sudo swift-get-nodes /etc/swift/container.ring.gz AUTH_2 CONTENTS

ssh "ls -lah /srv/node/sdb1/containers/2843/caf/02c6f2a7fedca02d0ed0e8636650ccaf/" ssh "ls -lah /srv/node/sdb1/containers/2843/caf/02c6f2a7fedca02d0ed0e8636650ccaf/" ssh "ls -lah /srv/node/sdb1/containers/2843/caf/02c6f2a7fedca02d0ed0e8636650ccaf/" ssh "ls -lah /srv/node/sdb1/containers/2843/caf/02c6f2a7fedca02d0ed0e8636650ccaf/"

In this case however none of the servers has the dbs file and paths.

Then i tried to grep for CONTENTS in the /srv/node/sdb1/containers and from the result i find that the container db is on the path


As you can see aside from the directory after the 'container' ./caf/02c6f2a7fedca02d0ed0e8636650ccaf/

As per title suggests what causes this path change after swift-ring rebalance?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2014-01-23 20:02:37 -0500

torgomatic gravatar image

Well, you can see the partition numbers in those paths; it's the number right after "/containers/".

swift-get-nodes is telling you that the container ring on this machine thinks the container has partition 2843; in binary, that's 101100011011, and if we left-pad with zeros to get 000000101100011011, we can see that it's the first 18 bits of 02c6f2a7fedca02d0ed0e8636650ccaf.

On the filesystem, it's got partition number 44, or 101100. Left-padding with the same six zeros as before, we get 000000101100, which is the first 12 bits of 02c6f2a7fedca02d0ed0e8636650ccaf.

This means that the ring's part_power was changed between when the container was created and when swift-get-nodes was run. NEVER DO THIS. When a ring's part_power changes, the partition number of every single entity of that type also changes, so all your old containers can't be found.

You can probably recover your old data by

  1. figuring out which rings had their part_powers changed
  2. recreating those rings with the correct part_power (12, here)
  3. waiting for replicators to move everything to the right place

However, if you've created any new entities (accounts, containers, or objects) while your cluster was running with the bad (part_power=18) rings, that new data will be inaccessible once you install the fixed (part_power=12) rings. If your cluster has both pre-goof and post-goof data in it, then it's time to pick which data set you love more.

edit flag offensive delete link more


Thank you for the answer, since its was development environment i did not pay attention to partition power and as you said i did recreate with 18. It was a good lesson to me.

fga gravatar imagefga ( 2014-01-24 00:34:02 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-01-23 01:31:13 -0500

Seen: 609 times

Last updated: Jan 23 '14