swift not replicating properly

asked 2015-10-14 05:46:22 -0500

Nelson Mota gravatar image

Hello everyone,

We have a openstack swift installation for development purposes, and we are facing problems with swift not replicating properly. Its supposed to save always 3 replicas. In my tests, swift places only one copy of the object, and sometimes two, but never 3.

ring configuration is:

[root@openstack3 ~]# swift-ring-builder /etc/swift/object.builder
/etc/swift/object.builder, build version 11 512 partitions, 3.000000 replicas, 2 regions, 3 zones, 5 devices, 0.98 balance, 100.00 dispersion

The minimum number of hours before a partition can be reassigned is 1

The overload factor is 0.00% (0.000000)

Devices: id region zone ip address port replication ip replication port name weight partitions balance meta

         0       1     1      10.10.6.21  6000      10.10.6.21              6000    mpatha 500.00       1160   -0.31
         1       1     1      10.10.6.11  6000      10.10.6.11              6000     disk1  40.00         94    0.98
         3       1     2      10.10.6.13  6000      10.10.6.13              6000     disk1  40.00         94    0.98
         4       2     1      10.10.6.14  6000      10.10.6.14              6000     disk1  40.00         94    0.98
         5       1     2      10.10.6.12  6000      10.10.6.12              6000     disk1  40.00         94    0.98

We have a MRC, with 2 regions, and when i place one big object (2Gb) and monitor disk spaces of nodes with df, we observed that it places several times the object in zones of the same region, and only two copys, and sometimes only one copy.

When i test the placement with swift-get-nodes i get this:

[root@swift-1 ~]# swift-get-nodes dummy3

Account dummy3

Container None

Object None

Partition 42 Hash ab5f330198b6e64e76ffada05c64d6ed

Server:Port Device 10.10.6.14:6002 disk1

Server:Port Device 10.10.6.21:6002 mpatha

Server:Port Device 10.10.6.13:6002 disk1 [Handoff]

Server:Port Device 10.10.6.12:6002 disk1 [Handoff]

curl -I -XHEAD "http://10.10.6.14:6002/disk1/42/dummy3" curl -I -XHEAD "http://10.10.6.21:6002/mpatha/42/dummy3" curl -I -XHEAD "http://10.10.6.13:6002/disk1/42/dummy3" # [Handoff] curl -I -XHEAD "http://10.10.6.12:6002/disk1/42/dummy3" # [Handoff]

Use your own device location of servers: such as "export DEVICE=/srv/node" ssh 10.10.6.14 "ls -lah ${DEVICE:-/srv/node}/disk1/accounts/42/6ed/ab5f330198b6e64e76ffada05c64d6ed" ssh 10.10.6.21 "ls -lah ${DEVICE:-/srv/node}/mpatha/accounts/42/6ed/ab5f330198b6e64e76ffada05c64d6ed" ssh 10.10.6.13 "ls -lah ${DEVICE:-/srv/node}/disk1/accounts/42/6ed/ab5f330198b6e64e76ffada05c64d6ed" # [Handoff] ssh 10.10.6.12 "ls -lah ${DEVICE:-/srv/node}/disk1/accounts/42/6ed/ab5f330198b6e64e76ffada05c64d6ed" # [Handoff]

note: /srv/node* is used as default value of devices, the real value is set in the config file on each storage node.

Any clues, whats going on?

Cheers!

edit retag flag offensive close merge delete