Ask Your Question
0

Increase swift object replication speed

asked 2018-01-03 08:40:32 -0600

VELYCHKOVSKY gravatar image

updated 2018-01-16 05:56:52 -0600

Hello. How I can to speed up object replication process ? I have few buckets with 5-10 million small objects inside. Then I've added new HDDs, start rebalancing and noticed that object-replicator speed is very slow and I need to wait 6000 hour for replication finish. At the same time I see that my HDDs doesn't expect very high IO load, and I think object-replicator proccess could use this capacity to speed up but it doesn't.

Is there some way to make replication more aggressive ?

This is my /etc/swift/object-server.conf

[DEFAULT]
bind_port = 6000
user = swift
swift_dir = /etc/swift
devices = /mnt/swift
mount_check = True
log_level = ERROR
conn_timeout = 5
container_update_timeout = 5
node_timeout = 5
max_clients = 4096

[pipeline:main]
pipeline = healthcheck recon object-server

[app:object-server]
use = egg:swift#object
replication_concurrency = 1500
replication_one_per_device = False
replication_lock_timeout = 30

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock

[object-replicator]
concurrency = 1500
run_pause = 5
interval = 5
log_level = DEBUG
stats_interval = 10
rsync_io_timeout = 60

[object-reconstructor]

[object-updater]
concurrency = 200
interval = 20
slowdown = 0.008
log_level = DEBUG

[object-auditor]
interval = 300


[filter:xprofile]
use = egg:swift#xprofile

/etc/rsyncd.conf

[object]
path            = /mnt/swift
read only       = false
write only      = no
list            = yes
incoming chmod  = 0644
outgoing chmod  = 0644
max connections = 1500
lock file =     /var/lock/object.lock
edit retag flag offensive close merge delete

Comments

Can anyone help with this ?

Jan 16 11:39:11 object-replicator: 9219/141930 (6.50%) partitions replicated in 1546362.86s (0.01/sec, 6183h remaining)

Looks like I must wait of my cluster rebalance ~1 year... this is very disappoint me..

VELYCHKOVSKY gravatar imageVELYCHKOVSKY ( 2018-01-16 05:56:26 -0600 )edit

1 answer

Sort by » oldest newest most voted
0

answered 2019-01-05 11:05:24 -0600

LeTonPhat gravatar image

Did you try these options in object-server.conf ?

replicator_workers = 0

concurrency = 1

i see it new in 2.18 . link: https://docs.openstack.org/releasenotes/swift/rocky.html (https://docs.openstack.org/releasenot...)

Add a multiprocess mode to the object replicator. Setting the replicator_workers setting to a positive value N will result in the replicator using up to N worker processes to perform replication tasks. At most one worker per disk will be spawned.

Worker process logs will have a bit of information prepended so operators can tell which messages came from which worker. The prefix is “[worker M/N pid=P] “, where M is the worker’s index, N is the total number of workers, and P is the process ID. Every message from the replicator’s logger will have the prefix

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2018-01-03 08:39:11 -0600

Seen: 578 times

Last updated: Jan 05