Ring Builder files lost swift

asked 2015-09-14 06:17:43 -0500

updated 2015-10-05 06:34:16 -0500

Hello We have an productive swift environment. After moving over the swift-proxies and deleting the old ones we found, that we have lost the builder files. The backups were only local, and I have to admit, I was very stupid. We use juno and swift 2.4.0 but the initial setup of the environment must have been with <= 1.7.5 The recovery with this help https://ask.openstack.org/en/question/20952/how-to-recovery-ring-builder-files/ (https://ask.openstack.org/en/question...) stops with

builder._set_parts_wanted()

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/dist-packages/swift/common/ring/builder.py", line 550, in _set_parts_wanted dev['parts']) Any help is highly apreciated.

Update:

The trouble seems to be, that the original files were built with swift <=1.7.5 The function _set_parts_wanted iters over the object.devs dictionaries and adds a parts_wanted key.

for dev in self._iter_devs():
            if not dev['weight']:

                dev['parts_wanted'] = -self.parts * self.replicas
            else:
                dev['parts_wanted'] = \
                    int(weight_of_one_part * dev['weight']) - dev['parts']

by setting the key parts to 0 I am able to restore the builder files.

So I end up with new builder files, and the actual content of the e.g. account.ring.gz seems to be the same as of the newly generated with swift-ring-builder account.builder write_ring.

My tests are

from swift.common.ring import RingData, RingBuilder
ring = RingData.load('old_account.ring.gz')
ring2 = RingData.load('new_account.ring.gz')
old = vars(ring)
new = vars(ring2)
if old == new:
     print "okay"
else:
     exit ()
if ring._replica2part2dev_id == ring2._replica2part2dev_id:
    print ring.devs[200]
    print "\n"
    print ring2.devs[200]
else:
    print "Error"

So everything seems to be alright. Still I feel a bit uncertain. Is there anything else I should observe/test? best regards Frank

New update: Thx to the great help of kota_ at IRC #openstack-swift I feel more certain about the 3 builder files. As I am on holiday right now, I'ld like to keep the question opened til next month when we will build an activate a new ring during a planned maintenance. Keep you on track here.

Final Update:

New ringfiles are in place, devices got removed and newly added, ring got rebalanced and spreaded. Everything is working as it should! So here is what worked for me

from swift.common.ring import RingData, RingBuilder
###Object from old ring
ring = RingData.load('/root/tools/ring/account.ring.gz')
import math

partitions = len(ring._replica2part2dev_id[0])
replicas = len(ring._replica2part2dev_id)
builder = RingBuilder(int(math.log(partitions, 2)), replicas, 1)

builder.devs = ring.devs
builder._replica2part2dev = ring._replica2part2dev_id
builder._last_part_moves_epoch = 0
from array import array

builder._last_part_moves = array('B', (0 for _ in xrange(partitions)))
### Here it is important to define the 'parts' key first !!!

    for d in builder._iter_devs():
                    d['parts'] = 0
    for p2d in builder._replica2part2dev:
        for dev_id in p2d:
            builder.devs[dev_id]['parts'] += 1
    builder._set_parts_wanted()


    builder.change_min_part_hours(24)
    builder.validate()
    import pickle
    pickle.dump(builder.to_dict(), open('account.builder', 'wb ...
(more)
edit retag flag offensive close merge delete