Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

My client sees these errors as the connection being closed before all data is received.

I don't understand what we would find out by adjusting node_timeout.

There are all kinds of errors. :-P I've pasted some recent ones below. The logs I pasted are for all servers. You can see immediately after the timestamp it shows the hostname of the server the data came from.

I've doubled the number of object workers from 8 to 16. I'll see if that reduces errors noticeably.

Here are some errors: swift105: Oct 17 14:24:32 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: txe1754390285a49c38cbd128aecaa5fe7) swift105: Oct 17 14:24:32 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: txf6c8946ffe464ee082891185f8d9d189) swift105: Oct 17 14:25:03 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d3 (saving for async update later): ConnectionTimeout (0.5s) (txn: txa90447e0ab4044ea95a9c0d03a0c53aa) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: tx10e193ec5c9c44a08a1918580b789d57) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: tx2695ba920505408eb91dd23faaeee349) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: tx51fdefef4dc744d99461d579e37a8d43) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: txd50e07b5f449486a884c24bf7852cf26) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: txed7ab47336ef4900b3bd4eb98598917f) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: txfed5461a2d644faa9b4c3aac5a3ddce5) swift105: Oct 17 14:26:35 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx3731beb2dedf46eabe748bae78e30c85) swift105: Oct 17 14:26:35 swift105 swift ERROR with Object server 172.17.10.73:6000/d2 re: Trying to DELETE /.expiring_objects/1381536000/1381541992-AUTH_software/rpmcache/afe96635c9f7f8372525a970511a64ba5eb0702f14a4664dc240045c7308a260.data: ConnectionTimeout (0.5s) (txn: txa7a32e4bafd44d83b23451a14c05bbd1) swift105: Oct 17 14:26:35 swift105 swift ERROR with Object server 172.17.10.75:6000/d4 re: Trying to DELETE /.expiring_objects/1381536000/1381541992-AUTH_software/rpmcache/afe96635c9f7f8372525a970511a64ba5eb0702f14a4664dc240045c7308a260.data: ConnectionTimeout (0.5s) (txn: txa7a32e4bafd44d83b23451a14c05bbd1) swift105: Oct 17 14:32:53 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx3a05ad050e124af4a56fa74b56b1e5df) swift105: Oct 17 14:32:53 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx8383669ad0ba44e3a155b1664b03840c) swift105: Oct 17 14:32:53 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: txc349537aaabf40739660aced0dbff76d) swift105: Oct 17 14:32:53 swift105 swift ERROR with Object server 172.17.10.75:6000/d4 re: Trying to DELETE /AUTH_software/rpmcache/2489a61277455bf04fa54a70a1951cac972ff2432385368530ef8643cbc3769c.data: ConnectionTimeout (0.5s) (txn: tx53bab8a03c024f52961be452c49a6c74) swift105: Oct 17 14:33:30 swift105 swift ERROR with Object server 172.17.10.74:6000/d2 re: Trying to DELETE /AUTH_software/rpmcache/e5f664503cef6505a83a3800f7ac9931c2c4a3d903939d2574617c3968ff7c01.data: Timeout (10s) (txn: txe3e566c46a0b4e8ca6a71fa18e230e91) swift105: Oct 17 14:35:16 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): Timeout (3s) (txn: tx32922ea664db4506be38ba3f2dd3fa61) swift105: Oct 17 14:35:16 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): Timeout (3s) (txn: tx386b22640365456cad2d8dd65929fc31) swift105: Oct 17 14:35:16 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): Timeout (3s) (txn: tx3a4675a91b4c457a932afe657b363afa) swift105: Oct 17 14:35:16 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): Timeout (3s) (txn: tx9663e9dc362f4ced88037fdd94dd213c) swift105: Oct 17 14:35:16 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): Timeout (3s) (txn: tx9a759de0f931476985ea448c007e0ae6) swift105: Oct 17 14:35:16 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): Timeout (3s) (txn: txfd19b6fe26694fd6a90c812b03c5558c) swift105: Oct 17 14:35:21 swift105 proxy-server ERROR with Object server 172.17.10.74:6000/d4 re: Trying to GET /v1/AUTH_software/buildcache/656a3c94d0f493f009b24311bafc06cb75eae672.data: Connection refused (txn: tx81922f0bb2574dfdb34763e79e1754d5) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:21 swift105 proxy-server ERROR with Object server 172.17.10.75:6000/d3 re: Trying to GET /v1/AUTH_software/buildcache/656a3c94d0f493f009b24311bafc06cb75eae672.data: Connection refused (txn: tx81922f0bb2574dfdb34763e79e1754d5) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:22 swift105 proxy-server ERROR with Object server 172.17.10.72:6000/d2 re: Trying to GET /v1/AUTH_software/buildcache/49bb3373fef456c8e69b615863fee15525d54840.data: Connection refused (txn: tx809280d50f924d9fb9b5f4ad06d51446) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:22 swift105 proxy-server ERROR with Object server 172.17.10.73:6000/d2 re: Trying to GET /v1/AUTH_software/buildcache/49bb3373fef456c8e69b615863fee15525d54840.data: Connection refused (txn: tx809280d50f924d9fb9b5f4ad06d51446) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:22 swift105 proxy-server ERROR with Object server 172.17.10.73:6000/d3 re: Trying to GET /v1/AUTH_software/buildcache/49bb3373fef456c8e69b615863fee15525d54840.data: Connection refused (txn: tx809280d50f924d9fb9b5f4ad06d51446) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:22 swift105 proxy-server ERROR with Object server 172.17.10.74:6000/d3 re: Trying to GET /v1/AUTH_software/buildcache/49bb3373fef456c8e69b615863fee15525d54840.data: Connection refused (txn: tx809280d50f924d9fb9b5f4ad06d51446) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:22 swift105 proxy-server ERROR with Object server 172.17.10.74:6000/d4 re: Trying to GET /v1/AUTH_software/buildcache/49bb3373fef456c8e69b615863fee15525d54840.data: Connection refused (txn: tx809280d50f924d9fb9b5f4ad06d51446) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:22 swift105 proxy-server ERROR with Object server 172.17.10.75:6000/d4 re: Trying to GET /v1/AUTH_software/buildcache/49bb3373fef456c8e69b615863fee15525d54840.data: Connection refused (txn: tx809280d50f924d9fb9b5f4ad06d51446) (client_ip: 172.22.34.14) swift105: Oct 17 14:35:23 swift105 swift ERROR with Object server 172.17.10.74:6000/d1 re: Trying to DELETE /AUTH_software/rpmcache/1af7f2e3a0e1244aa812becb5d3e2b455b5c57864ba44eca824b11f7e555d654.data: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift-1.8.0-py2.7.egg/swift/proxy/controllers/base.py", line 581, in _make_request#012 resp = conn.getresponse()#012 File "/usr/lib/python2.7/site-packages/swift-1.8.0-py2.7.egg/swift/common/bufferedhttp.py", line 102, in getresponse#012 response = HTTPConnection.getresponse(self)#012 File "/usr/lib64/python2.7/httplib.py", line 1025, in getresponse#012 response.begin()#012 File "/usr/lib64/python2.7/httplib.py", line 401, in begin#012 version, status, reason = self._read_status()#012 File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status#012 raise BadStatusLine(line)#012BadStatusLine: '' (txn: txd21637df62a64d94858c62367c25c522) swift105: Oct 17 14:35:23 swift105 swift ERROR with Object server 172.17.10.74:6000/d2 re: Trying to DELETE /.expiring_objects/1381536000/1381542065-AUTH_software/rpmcache/1af7f2e3a0e1244aa812becb5d3e2b455b5c57864ba44eca824b11f7e555d654.data: Connection refused (txn: tx91ede11eebde4c4db8a07fdf3af30938) swift105: Oct 17 14:35:26 swift105 proxy-server ERROR with Object server 172.17.10.76:6000/d4 re: Trying to GET /v1/AUTH_software/buildcache/d6e6ff660c1adeb54963583af5321fb57acd24c8.data: Connection refused (txn: txeb409b7250ec4e2da75fb7e5cbc2d5ed) (client_ip: 172.22.34.14) swift105: Oct 17 14:37:22 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): ConnectionTimeout (0.5s) (txn: txfaefbe109a7e4942ae029912dbb72a0c) swift105: Oct 17 14:37:59 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx774bc9c87a6744d39b81e5c4a4dc711f) swift105: Oct 17 14:38:07 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): Timeout (3s) (txn: tx2ac418e7f059438a987b01af456f98ff) swift105: Oct 17 14:38:07 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): Timeout (3s) (txn: txbc9d6d794374430583639214c9346a16) swift105: Oct 17 14:38:07 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): Timeout (3s) (txn: txcb6e00ac81034a0593fd4beff5741be8) swift105: Oct 17 14:39:04 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: tx4513cfc938114716aa3c228d6555c22b) swift105: Oct 17 14:39:04 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: tx5967aa7462c8477fa071c3e908d3718a) swift105: Oct 17 14:39:04 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: tx5984822f6d104ae2913c1d99eb7a36ef) swift105: Oct 17 14:39:04 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: tx74b73ded1f434930a91d019b78e6ebe7) swift105: Oct 17 14:39:04 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: txe49f3d0887f34d409b4bb00befcc1f1a) swift105: Oct 17 14:41:14 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx40ce176d5ec24a5f9aadd3ce9c30da1a) swift105: Oct 17 14:41:14 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx614343c5225349979f494d77f90a817e) swift105: Oct 17 14:41:14 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx7f8f8b4be943476597b0c4b932b9a2f4) swift105: Oct 17 14:41:15 swift105 proxy-server ERROR with Container server 172.17.10.75:6001/d3 re: Trying to HEAD /v1/AUTH_software/check_swift: ConnectionTimeout (0.5s) (txn: tx0245928bc3b94afcaab6e1c7979da3f1) (client_ip: 172.22.22.37) swift105: Oct 17 14:41:52 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d4 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx7bdf71c4345a494f9906aac3a1dbb4ac) swift105: Oct 17 14:41:52 swift105 object-updater ERROR with remote server 172.17.10.72:6001/d2: ConnectionTimeout (0.5s) swift105: Oct 17 14:41:53 swift105 object-updater ERROR with remote server 172.17.10.73:6001/d4: ConnectionTimeout (0.5s) swift105: Oct 17 14:41:53 swift105 object-updater ERROR with remote server 172.17.10.74:6001/d4: ConnectionTimeout (0.5s) swift105: Oct 17 14:43:58 swift105 object-server ERROR container update failed with 172.17.10.71:6001/d3 (saving for async update later): ConnectionTimeout (0.5s) (txn: txe3a63d0dcff34d449250731b88124c3c) swift105: Oct 17 14:44:09 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d3 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx8b5b2b4571384684bb97d46f875911ac) swift105: Oct 17 14:47:41 swift105 proxy-server ERROR with Object server 172.17.10.72:6000/d2 re: Trying to GET /v1/AUTH_software/buildcache/299e07e6317262bb0529268ab85533340706d72c.data: ConnectionTimeout (0.5s) (txn: tx68b2f4f8453a4c6fa1976184903d4bd3) (client_ip: 172.22.34.14) swift105: Oct 17 14:47:41 swift105 proxy-server ERROR with Object server 172.17.10.73:6000/d1 re: Trying to GET /v1/AUTH_software/buildcache/299e07e6317262bb0529268ab85533340706d72c.data: ConnectionTimeout (0.5s) (txn: tx68b2f4f8453a4c6fa1976184903d4bd3) (client_ip: 172.22.34.14) swift105: Oct 17 14:48:09 swift105 swift ERROR with Object server 172.17.10.75:6000/d3 re: Trying to DELETE /.expiring_objects/1381536000/1381542293-AUTH_software/rpmcache/3848fd46c4a91be1a111b4b8c2b4826db46b5fdd778fdcabf1ed4ca9bc634426.data: Timeout (10s) (txn: tx5e12a0ccc3c349ae92774d1ede360b80)