Why is Swift not distributing files evenly between zones/servers?

asked 2016-03-10 13:50:51 -0500

chrisbzc gravatar image

I currently have a Swift cluster with three storage nodes with 4 devices each, with each node configured as a different zone and set to three replicas. I would expect this to mean that Swift would store one replica on each node, but for some reason that's not happening with all objects. Some of them have two replicas on the same node, for example:

sudo swift-get-nodes -a /etc/swift/object.ring.gz AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc

Account     AUTH_884…b02
Container   comms-attachments
Object      314/31484/3148479_KWQZycWvY2bB.doc


Partition   91623
Hash        5979e47465687e1f608470503c7de3ff

Server:Port Device  xx.xx.123.178:6000 4
Server:Port Device  xx.xx.123.178:6000 2
Server:Port Device  xx.xx.123.177:6000 2
Server:Port Device  xx.xx.123.176:6000 3     [Handoff]
Server:Port Device  xx.xx.123.177:6000 3     [Handoff]
Server:Port Device  xx.xx.123.177:6000 4     [Handoff]
Server:Port Device  xx.xx.123.176:6000 1     [Handoff]
Server:Port Device  xx.xx.123.177:6000 1     [Handoff]
Server:Port Device  xx.xx.123.176:6000 4     [Handoff]
Server:Port Device  xx.xx.123.176:6000 2     [Handoff]
Server:Port Device  xx.xx.123.178:6000 1     [Handoff]
Server:Port Device  xx.xx.123.178:6000 3     [Handoff]


curl -I -XHEAD "http://xx.xx.123.178:6000/4/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc"
curl -I -XHEAD "http://xx.xx.123.178:6000/2/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc"
curl -I -XHEAD "http://xx.xx.123.177:6000/2/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc"
curl -I -XHEAD "http://xx.xx.123.176:6000/3/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.177:6000/3/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.177:6000/4/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.176:6000/1/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.177:6000/1/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.176:6000/4/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.176:6000/2/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.178:6000/1/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]
curl -I -XHEAD "http://xx.xx.123.178:6000/3/91623/AUTH_884…b02/comms-attachments/314/31484/3148479_KWQZycWvY2bB.doc" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh xx.xx.123.178 "ls -lah ${DEVICE:-/srv/node*}/4/objects/91623/3ff/5979e47465687e1f608470503c7de3ff"
ssh xx.xx.123.178 "ls -lah ${DEVICE:-/srv/node*}/2/objects/91623/3ff/5979e47465687e1f608470503c7de3ff"
ssh xx.xx.123.177 "ls -lah ${DEVICE:-/srv/node*}/2/objects/91623/3ff/5979e47465687e1f608470503c7de3ff"
ssh xx.xx.123.176 "ls ...
(more)
edit retag flag offensive close merge delete