Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

update swift ring

Hi, I've already update my ring (just object, i add one new disk to the ring) and copied proper file(object builder and .gz file) to all nodes but the node doesnt rsync with other nodes for last object and just new object place in it this is journalctl log of my node:

``` swift-storage-2 object-replicator[2070]: Error syncing with node: {'index': 2, u'replication_port': 6206, u'weight': 6.0, u'z n_ip': u'192.168.200.212', u'meta': u'', u'device': u'sdd', u'port': 6202}: Timeout (60s)

swift-storage-02 object-server[2143]: ERROR container update failed with 172.20.40.253:6100/sdg (saving for async update later swift-storage-02 object-replicator[2070]: Error syncing with node: {'index': 2, u'replication_port': 6208, u'weight': 6.0, u'z swift-storage-02 account-replicator[2079]: Can't find itself 172.20.40.238 with port 6000 in ring file, not replicating

```

update swift ring

Hi, I've already update my ring (just object, i add one new disk to the ring) and copied proper file(object builder and .gz file) to all nodes but the node doesnt rsync with other nodes for last object and just new object place in it this is journalctl log of my node:

``` swift-storage-2 object-replicator[2070]: Apr 09 17:34:00 swift-storage-09 object-server[41657]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: txb76e2bafd6c04afa8e9ea-005e8f1d3f) Apr 09 17:34:00 swift-storage-09 object-server[41658]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: txa9d01077c5bb4a41b4a24-005e8f1d3f) Apr 09 17:34:04 swift-storage-09 object-server[41670]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: tx37ae98d018044c3a8dc56-005e8f1d43) Apr 09 17:34:04 swift-storage-09 object-server[41669]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: tx588cedbc0e104f1db64e8-005e8f1d43) Apr 09 17:34:04 swift-storage-09 object-server[41661]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: txedf59d36e5584b0a9ecf8-005e8f1d43) Apr 09 17:34:05 swift-storage-09 object-server[41660]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: tx25ca87b9d5614949b2182-005e8f1d44) Apr 09 17:34:09 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:10 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:10 swift-storage-09 object-server[41670]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: txebf956b16ac8461091bd4-005e8f1d4a) Apr 09 17:34:11 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:12 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:12 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: Host unreachable Apr 09 17:34:12 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:13 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:13 swift-storage-09 object-server[41659]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): ConnectionTimeout (0.5s) (txn: tx1e7f68030c1449ae9abd1-005e8f1d4c) Apr 09 17:34:13 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:14 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:14 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: ConnectionTimeout (0.5s) Apr 09 17:34:15 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: Host unreachable Apr 09 17:34:15 swift-storage-09 object-server[41666]: ERROR container update failed with 192.168.200.236:6100/sdl (saving for async update later): Host unreachable (txn: txc34458a9857f4fbfab6ab-005e8f1d4e) Apr 09 17:34:15 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: Connection refused Apr 09 17:34:15 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: Connection refused Apr 09 17:34:15 swift-storage-09 object-updater[55756]: ERROR with remote server 192.168.200.236:6100/sdl: Connection refused Apr 09 17:34:17 swift-storage-09 container-replicator[2098]: Can't find itself 192.168.200.238 with port 6100 in ring file, not replicating Apr 09 17:34:18 swift-storage-09 object-replicator[41637]: Error syncing with node: {'index': 2, u'replication_port': 6201, u'weight': 6.0, u'zone': 1, u'ip': u'192.168.200.231', u'region': 1, u'id': 11, u'replication_ip': u'192.168.200.231', u'meta': u'', u'device': u'sdc', u'port': 6201}: Timeout (60s) Apr 09 17:34:24 swift-storage-09 account-replicator[2091]: Can't find itself 192.168.200.238 with port 6000 in ring file, not replicating Apr 09 17:34:30 swift-storage-09 object-replicator[41637]: Error syncing with node: {'index': 1, u'replication_port': 6201, u'weight': 3.0, u'zone': 2, u'ip': u'192.168.200.251', u'region': 1, u'id': 139, u'replication_ip': u'192.168.200.251', u'meta': u'', u'device': u'sdc', u'port': 6201}: Timeout (60s) Apr 09 17:34:37 swift-storage-09 object-replicator[41637]: Error syncing with node: {'index': 1, u'replication_port': 6206, u'weight': 6.0, u'z n_ip': u'192.168.200.212', u'zone': 1, u'ip': u'192.168.200.230', u'region': 1, u'id': 6, u'replication_ip': u'192.168.200.230', u'meta': u'', u'device': u'sdh', u'port': 6206}: Timeout (60s) Apr 09 17:34:45 swift-storage-09 object-replicator[41637]: Error syncing with node: {'index': 1, u'replication_port': 6203, u'weight': 6.0, u'zone': 2, u'ip': u'192.168.200.237', u'region': 1, u'id': 73, u'replication_ip': u'192.168.200.237', u'meta': u'', u'device': u'sde', u'port': 6203}: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 441, in update#012 '', headers=headers).getresponse()#012 File "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line 175, in getresponse#012 response = HTTPConnection.getresponse(self)#012 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse#012 response.begin()#012 File "/usr/lib/python2.7/httplib.py", line 453, in begin#012 version, status, reason = self._read_status()#012 File "/usr/lib/python2.7/httplib.py", line 417, in _read_status#012 raise BadStatusLine(line)#012BadStatusLine: '' Apr 09 17:34:47 swift-storage-09 container-replicator[2098]: Can't find itself 192.168.200.238 with port 6100 in ring file, not replicating Apr 09 17:34:54 swift-storage-09 account-replicator[2091]: Can't find itself 192.168.200.238 with port 6000 in ring file, not replicating Apr 09 17:34:56 swift-storage-09 object-replicator[41637]: Error syncing with node: {'index': 1, u'replication_port': 6202, u'weight': 6.0, u'zone': 1, u'ip': u'192.168.200.230', u'region': 1, u'id': 2, u'replication_ip': u'192.168.200.230', u'meta': u'', u'device': u'sdd', u'port': 6202}: Timeout (60s)(60s) Apr 09 17:35:00 swift-storage-09 object-replicator[41637]: <f+++++++++ 030="" 8392f09975b2a87388cc6084e5dde030="" 1586435173.94532.data="" apr="" 09="" 17:35:00="" swift-storage-09="" object-replicator[41637]:="" <f+++++++++="" 084="" 8393bc731f3c4e57e013fe8049eb4084="" 1586436843.89448.data="" apr="" 09="" 17:35:00="" swift-storage-09="" object-replicator[41637]:="" <f+++++++++="" 3c7="" 83951ca32de39c7ab00eb85e1829c3c7="" 1586427318.50399.data="" apr="" 09="" 17:35:00="" swift-storage-09="" object-replicator[41637]:="" <f+++++++++="" a23="" 839677db729b233c08f8316259961a23="" 1586436809.74215.data="" apr="" 09="" 17:35:00="" swift-storage-09="" object-replicator[41637]:="" <f+++++++++="" aa5="" 83967c3565e9694f5e7cdd98b4a7daa5="" 1586433696.94126.data="" apr="" 09="" 17:35:00="" swift-storage-09="" object-replicator[41637]:="" <f+++++++++="" ca4="" 839062b2e09ae9bb1e91c3d7def65ca4="" 1586436856.57256.data="" apr="" 09="" 17:35:00="" swift-storage-09="" object-replicator[41637]:="" successful="" rsync="" of="" srv="" node="" sdb="" objects="" 4210="" f72="" at="" 192.168.200.230::object="" sdd="" objects="" 4210="" (139.268)<="" p="">

swift-storage-02 object-server[2143]: ERROR container update failed with 172.20.40.253:6100/sdg (saving for async update later swift-storage-02 object-replicator[2070]: Error syncing with node: {'index': 2, u'replication_port': 6208, u'weight': 6.0, u'z swift-storage-02 account-replicator[2079]: Can't find itself 172.20.40.238 with port 6000 in ring file, not replicating

```