Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Hi! When I am executing "swift stat" command I am getting Account HEAD failed: http://controller:8080/v1/AUTH_3babe56d610847c389f89c959f74dd44 503 Service Unavailable Failed Transaction ID: tx635b33ff8b4545afae451-00589b99db

Hi All, I have installed two swift storage node with IP 192.168.1.194 & 192.168.1.195. I have one proxy-server node as 192.168.1.191. When I executing swift stat command then I am getting Account HEAD failed: http://controller:8080/v1/AUTH_3babe56d610847c389f89c959f74dd44 503 Service Unavailable Failed Transaction ID: tx635b33ff8b4545afae451-00589b99db error. I have also checked with replacing swift storage IP by 0.0.0.0.0 in account-server.conf, object-server.conf and container-server.conf files.I have also checked with all ports. But still I am getting this error. Can anyone help me??

Here I am posting logs from proxy node(/var/log/message)

Feb 9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: txb3328471f8bb4feca2cec-00589b999b) Feb 9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: txb3328471f8bb4feca2cec-00589b999b) Feb 9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: txb3328471f8bb4feca2cec-00589b999b) Feb 9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: txb3328471f8bb4feca2cec-00589b999b) Feb 9 03:50:11 controller swift: Account HEAD returning 503 for [507, 507, 507, 507] (txn: txb3328471f8bb4feca2cec-00589b999b) Feb 9 03:50:11 controller swift: - - 08/Feb/2017/22/20/11 GET /v1/.misplaced_objects/1486591200%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txb3328471f8bb4feca2cec-00589b999b - 0.0335 - - 1486592411.740717888 1486592411.774223089 - Feb 9 03:50:11 controller swift: Account GET returning 503 for [] (txn: tx4c71cc5165a3426180e12-00589b999b) Feb 9 03:50:13 controller swift: Account GET returning 503 for [] (txn: txdbeb707711ba4d0fa2b1a-00589b999d) Feb 9 03:50:13 controller swift: - - 08/Feb/2017/22/20/13 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx4c71cc5165a3426180e12-00589b999b - 2.0105 - - 1486592411.775178909 1486592413.785631895 - (txn: txdbeb707711ba4d0fa2b1a-00589b999d) Feb 9 03:50:17 controller swift: Account GET returning 503 for [] (txn: tx8b1c640bc6b1406e95a6d-00589b99a1) Feb 9 03:50:17 controller swift: - - 08/Feb/2017/22/20/17 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - txdbeb707711ba4d0fa2b1a-00589b999d - 4.0123 - - 1486592413.780395031 1486592417.792692900 - (txn: tx8b1c640bc6b1406e95a6d-00589b99a1) Feb 9 03:50:17 controller container-reconciler: Error listing containers in account .misplaced_objects (Unexpected response: 503 Service Unavailable) (txn: tx8b1c640bc6b1406e95a6d-00589b99a1) Feb 9 03:50:17 controller swift: - - 08/Feb/2017/22/20/17 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx8b1c640bc6b1406e95a6d-00589b99a1 - 0.0060 - - 1486592417.787771940 1486592417.793772936 - Feb 9 03:50:17 controller container-reconciler: Reconciler Stats: {} (txn: tx8b1c640bc6b1406e95a6d-00589b99a1) Feb 9 03:50:17 controller container-reconciler: sleeping between intervals (30s) (txn: tx8b1c640bc6b1406e95a6d-00589b99a1) Feb 9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 129, in get#012 fp, sock = super(MemcacheConnPool, self).get()#012 File "/usr/lib/python2.7/site-packages/eventlet/pools.py", line 88, in get#012 created = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: Deferring reject downstream Feb 9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: Account HEAD returning 503 for [507, 507, 507, 507] (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: - - 08/Feb/2017/22/20/43 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - txfb09485fa182402093253-00589b99bb - 0.0121 RL - 1486592443.282943010 1486592443.295042038 - Feb 9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb) Feb 9 03:50:43 controller proxy-server: Account HEAD returning 503 for [] (txn: txfb09485fa182402093253-00589b99bb) (client_ip: 192.168.1.191) Feb 9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb) (client_ip: 192.168.1.191) Feb 9 03:50:43 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/43 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - txfb09485fa182402093253-00589b99bb - 0.3487 - - 1486592443.280867100 1486592443.629539967 - Feb 9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc) (client_ip: 192.168.1.191) Feb 9 03:50:44 controller proxy-server: Deferring reject downstream Feb 9 03:50:44 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0096bc7625e1444baed66-00589b99bc) Feb 9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc) Feb 9 03:50:44 controller proxy-server: - - 08/Feb/2017/22/20/44 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx0096bc7625e1444baed66-00589b99bc - 0.0121 RL - 1486592444.642867088 1486592444.655015945 - Feb 9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc) Feb 9 03:50:44 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0096bc7625e1444baed66-00589b99bc) (client_ip: 192.168.1.191) Feb 9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc) (client_ip: 192.168.1.191) Feb 9 03:50:44 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/44 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx0096bc7625e1444baed66-00589b99bc - 0.2538 - - 1486592444.637703896 1486592444.891524076 - Feb 9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txf905851da78b4ca9be972-00589b99be) (client_ip: 192.168.1.191) Feb 9 03:50:46 controller proxy-server: Deferring reject downstream Feb 9 03:50:46 controller proxy-server: Account HEAD returning 503 for [] (txn: txf905851da78b4ca9be972-00589b99be) Feb 9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txf905851da78b4ca9be972-00589b99be) Feb 9 03:50:46 controller proxy-server: - - 08/Feb/2017/22/20/46 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - txf905851da78b4ca9be972-00589b99be - 0.0046 RL - 1486592446.903759003 1486592446.908375978 - Feb 9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txf905851da78b4ca9be972-00589b99be) Feb 9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error limiting server controller:11211 (txn: txf905851da78b4ca9be972-00589b99be) Feb 9 03:50:47 controller proxy-server: Account HEAD returning 503 for [] (txn: txf905851da78b4ca9be972-00589b99be) (client_ip: 192.168.1.191) Feb 9 03:50:47 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/47 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - txf905851da78b4ca9be972-00589b99be - 0.2442 - - 1486592446.899775028 1486592447.143946886 - Feb 9 03:50:47 controller swift: Account HEAD returning 503 for [] (txn: txb2fa7b4f670d4860b6acc-00589b99bf) Feb 9 03:50:47 controller swift: - - 08/Feb/2017/22/20/47 GET /v1/.misplaced_objects/1486591200%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txb2fa7b4f670d4860b6acc-00589b99bf - 0.0085 - - 1486592447.826589108 1486592447.835084915 - Feb 9 03:50:47 controller swift: Account GET returning 503 for [] (txn: tx0979b5d107d14435bed4b-00589b99bf) Feb 9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0) Feb 9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0) Feb 9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0) Feb 9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0) Feb 9 03:50:48 controller swift: Account HEAD returning 503 for [507, 507, 507, 507] (txn: tx35d8c441a5fe4cefb1eda-00589b99c0) Feb 9 03:50:49 controller swift: Account GET returning 503 for [] (txn: txd8343cd08e9840eeb59df-00589b99c1) Feb 9 03:50:49 controller swift: - - 08/Feb/2017/22/20/49 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx0979b5d107d14435bed4b-00589b99bf - 2.0123 - - 1486592447.836558104 1486592449.848830938 - (txn: txd8343cd08e9840eeb59df-00589b99c1) Feb 9 03:50:50 controller swift: Account HEAD returning 503 for [] (txn: txa62279ce22c5442c86935-00589b99c2) Feb 9 03:50:51 controller proxy-server: Deferring reject downstream Feb 9 03:50:51 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0e68587b841145abb2457-00589b99c3) Feb 9 03:50:51 controller proxy-server: - - 08/Feb/2017/22/20/51 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx0e68587b841145abb2457-00589b99c3 - 0.0031 RL - 1486592451.151578903 1486592451.154645920 - Feb 9 03:50:51 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0e68587b841145abb2457-00589b99c3) (client_ip: 192.168.1.191) Feb 9 03:50:51 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/51 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx0e68587b841145abb2457-00589b99c3 - 0.2539 - - 1486592451.149534941 1486592451.403433084 - Feb 9 03:50:53 controller swift: Account GET returning 503 for [] (txn: tx59094c241e584f848c732-00589b99c5) Feb 9 03:50:53 controller swift: - - 08/Feb/2017/22/20/53 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - txd8343cd08e9840eeb59df-00589b99c1 - 4.0157 - - 1486592449.843744040 1486592453.859456062 - (txn: tx59094c241e584f848c732-00589b99c5) Feb 9 03:50:53 controller container-reconciler: Error listing containers in account .misplaced_objects (Unexpected response: 503 Service Unavailable) (txn: tx59094c241e584f848c732-00589b99c5) Feb 9 03:50:53 controller swift: - - 08/Feb/2017/22/20/53 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx59094c241e584f848c732-00589b99c5 - 0.0072 - - 1486592453.853568077 1486592453.860737085 - Feb 9 03:50:53 controller container-reconciler: Reconciler Stats: {} (txn: tx59094c241e584f848c732-00589b99c5) Feb 9 03:50:53 controller container-reconciler: sleeping between intervals (30s) (txn: tx59094c241e584f848c732-00589b99c5) Feb 9 03:50:54 controller swift: Account HEAD returning 503 for [] (txn: tx41ce67418b754a8ea9f22-00589b99c6) Feb 9 03:50:54 controller object-expirer: Unhandled exception: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 172, in run_once#012 self.swift.get_account_info(self.expiring_objects_account)#012 File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 361, in get_account_info#012 resp = self.make_request('HEAD', path, {}, acceptable_statuses)#012 File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 195, in make_request#012 _('Unexpected response: %s') % resp.status, resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn: tx41ce67418b754a8ea9f22-00589b99c6) Feb 9 03:50:59 controller proxy-server: Deferring reject downstream Feb 9 03:50:59 controller proxy-server: Account HEAD returning 503 for [] (txn: tx4ae4184315c74bd6acc70-00589b99cb) Feb 9 03:50:59 controller proxy-server: - - 08/Feb/2017/22/20/59 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx4ae4184315c74bd6acc70-00589b99cb - 0.0028 RL - 1486592459.413984060 1486592459.416832924 - Feb 9 03:50:59 controller proxy-server: Account HEAD returning 503 for [] (txn: tx4ae4184315c74bd6acc70-00589b99cb) (client_ip: 192.168.1.191) Feb 9 03:50:59 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/59 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx4ae4184315c74bd6acc70-00589b99cb - 0.2364 - - 1486592459.411684990 1486592459.648108006 - Feb 9 03:51:15 controller proxy-server: Deferring reject downstream Feb 9 03:51:15 controller proxy-server: Account HEAD returning 503 for [] (txn: tx635b33ff8b4545afae451-00589b99db) Feb 9 03:51:15 controller proxy-server: - - 08/Feb/2017/22/21/15 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx635b33ff8b4545afae451-00589b99db - 0.0027 RL - 1486592475.671029091 1486592475.673742056 - Feb 9 03:51:15 controller proxy-server: Account HEAD returning 503 for [] (txn: tx635b33ff8b4545afae451-00589b99db) (client_ip: 192.168.1.191) Feb 9 03:51:15 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/21/15 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx635b33ff8b4545afae451-00589b99db - 0.2344 - - 1486592475.669040918 1486592475.903404951 - Feb 9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3) Feb 9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3) Feb 9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3) Feb 9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3) Feb 9 03:51:23 controller swift: Account HEAD returning 503 for [507, 507, 507, 507] (txn: txe4e9a293b2be4d9ea5f62-00589b99e3) Feb 9 03:51:23 controller swift: - - 08/Feb/2017/22/21/23 GET /v1/.misplaced_objects/1486591200%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txe4e9a293b2be4d9ea5f62-00589b99e3 - 0.0346 - - 1486592483.881172895 1486592483.915751934 - Feb 9 03:51:23 controller swift: Account GET returning 503 for [] (txn: tx3a6089ddf65842b2ad7a1-00589b99e3) Feb 9 03:51:25 controller swift: Account GET returning 503 for [] (txn: tx85f3f706856748e9a8b58-00589b99e5) Feb 9 03:51:25 controller swift: - - 08/Feb/2017/22/21/25 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx3a6089ddf65842b2ad7a1-00589b99e3 - 2.0101 - - 1486592483.916512966 1486592485.926599979 - (txn: tx85f3f706856748e9a8b58-00589b99e5) Feb 9 03:51:29 controller swift: Account GET returning 503 for [] (txn: tx27733d4774f940879eb3b-00589b99e9) Feb 9 03:51:29 controller swift: - - 08/Feb/2017/22/21/29 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx85f3f706856748e9a8b58-00589b99e5 - 4.0149 - - 1486592485.920792103 1486592489.935691118 - (txn: tx27733d4774f940879eb3b-00589b99e9) Feb 9 03:51:29 controller container-reconciler: Error listing containers in account .misplaced_objects (Unexpected response: 503 Service Unavailable) (txn: tx27733d4774f940879eb3b-00589b99e9) Feb 9 03:51:29 controller swift: - - 08/Feb/2017/22/21/29 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx27733d4774f940879eb3b-00589b99e9 - 0.0062 - - 1486592489.930593967 1486592489.936772108 - Feb 9 03:51:29 controller container-reconciler: Reconciler Stats: {} (txn: tx27733d4774f940879eb3b-00589b99e9) Feb 9 03:51:29 controller container-reconciler: sleeping between intervals (30s) (txn: tx27733d4774f940879eb3b-00589b99e9)

Hi! When I am executing "swift stat" command I am getting Account HEAD failed: http://controller:8080/v1/AUTH_3babe56d610847c389f89c959f74dd44 503 Service Unavailable Failed Transaction ID: tx635b33ff8b4545afae451-00589b99db

Hi All, All,

I have installed two swift storage node with IP 192.168.1.194 & 192.168.1.195. I have one proxy-server node as 192.168.1.191. When I executing swift stat command then I am getting

Account HEAD failed: http://controller:8080/v1/AUTH_3babe56d610847c389f89c959f74dd44 503 Service Unavailable
Failed Transaction ID: tx635b33ff8b4545afae451-00589b99db tx635b33ff8b4545afae451-00589b99db

error. I have also checked with replacing swift storage IP by 0.0.0.0.0 in account-server.conf, object-server.conf and container-server.conf files.I have also checked with all ports. But still I am getting this error. Can anyone help me??

Here I am posting logs from proxy node(/var/log/message)

Feb  9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: txb3328471f8bb4feca2cec-00589b999b)
Feb  9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: txb3328471f8bb4feca2cec-00589b999b)
Feb  9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: txb3328471f8bb4feca2cec-00589b999b)
Feb  9 03:50:11 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: txb3328471f8bb4feca2cec-00589b999b)
Feb  9 03:50:11 controller swift: Account HEAD returning 503 for [507, 507, 507, 507] (txn: txb3328471f8bb4feca2cec-00589b999b)
Feb  9 03:50:11 controller swift: - - 08/Feb/2017/22/20/11 GET /v1/.misplaced_objects/1486591200%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txb3328471f8bb4feca2cec-00589b999b - 0.0335 - - 1486592411.740717888 1486592411.774223089 -
Feb  9 03:50:11 controller swift: Account GET returning 503 for [] (txn: tx4c71cc5165a3426180e12-00589b999b)
Feb  9 03:50:13 controller swift: Account GET returning 503 for [] (txn: txdbeb707711ba4d0fa2b1a-00589b999d)
Feb  9 03:50:13 controller swift: - - 08/Feb/2017/22/20/13 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx4c71cc5165a3426180e12-00589b999b - 2.0105 - - 1486592411.775178909 1486592413.785631895 - (txn: txdbeb707711ba4d0fa2b1a-00589b999d)
Feb  9 03:50:17 controller swift: Account GET returning 503 for [] (txn: tx8b1c640bc6b1406e95a6d-00589b99a1)
Feb  9 03:50:17 controller swift: - - 08/Feb/2017/22/20/17 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - txdbeb707711ba4d0fa2b1a-00589b999d - 4.0123 - - 1486592413.780395031 1486592417.792692900 - (txn: tx8b1c640bc6b1406e95a6d-00589b99a1)
Feb  9 03:50:17 controller container-reconciler: Error listing containers in account .misplaced_objects (Unexpected response: 503 Service Unavailable) (txn: tx8b1c640bc6b1406e95a6d-00589b99a1)
Feb  9 03:50:17 controller swift: - - 08/Feb/2017/22/20/17 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx8b1c640bc6b1406e95a6d-00589b99a1 - 0.0060 - - 1486592417.787771940 1486592417.793772936 -
Feb  9 03:50:17 controller container-reconciler: Reconciler Stats: {} (txn: tx8b1c640bc6b1406e95a6d-00589b99a1)
Feb  9 03:50:17 controller container-reconciler: sleeping between intervals (30s) (txn: tx8b1c640bc6b1406e95a6d-00589b99a1)
Feb  9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 129, in get#012    fp, sock = super(MemcacheConnPool, self).get()#012  File "/usr/lib/python2.7/site-packages/eventlet/pools.py", line 88, in get#012    created = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: Deferring reject downstream
Feb  9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: Account HEAD returning 503 for [507, 507, 507, 507] (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: - - 08/Feb/2017/22/20/43 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - txfb09485fa182402093253-00589b99bb - 0.0121 RL - 1486592443.282943010 1486592443.295042038 -
Feb  9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb)
Feb  9 03:50:43 controller proxy-server: Account HEAD returning 503 for [] (txn: txfb09485fa182402093253-00589b99bb) (client_ip: 192.168.1.191)
Feb  9 03:50:43 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txfb09485fa182402093253-00589b99bb) (client_ip: 192.168.1.191)
Feb  9 03:50:43 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/43 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - txfb09485fa182402093253-00589b99bb - 0.3487 - - 1486592443.280867100 1486592443.629539967 -
Feb  9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc) (client_ip: 192.168.1.191)
Feb  9 03:50:44 controller proxy-server: Deferring reject downstream
Feb  9 03:50:44 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0096bc7625e1444baed66-00589b99bc)
Feb  9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc)
Feb  9 03:50:44 controller proxy-server: - - 08/Feb/2017/22/20/44 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx0096bc7625e1444baed66-00589b99bc - 0.0121 RL - 1486592444.642867088 1486592444.655015945 -
Feb  9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc)
Feb  9 03:50:44 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0096bc7625e1444baed66-00589b99bc) (client_ip: 192.168.1.191)
Feb  9 03:50:44 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx0096bc7625e1444baed66-00589b99bc) (client_ip: 192.168.1.191)
Feb  9 03:50:44 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/44 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx0096bc7625e1444baed66-00589b99bc - 0.2538 - - 1486592444.637703896 1486592444.891524076 -
Feb  9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txf905851da78b4ca9be972-00589b99be) (client_ip: 192.168.1.191)
Feb  9 03:50:46 controller proxy-server: Deferring reject downstream
Feb  9 03:50:46 controller proxy-server: Account HEAD returning 503 for [] (txn: txf905851da78b4ca9be972-00589b99be)
Feb  9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txf905851da78b4ca9be972-00589b99be)
Feb  9 03:50:46 controller proxy-server: - - 08/Feb/2017/22/20/46 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - txf905851da78b4ca9be972-00589b99be - 0.0046 RL - 1486592446.903759003 1486592446.908375978 -
Feb  9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error connecting to memcached: controller:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txf905851da78b4ca9be972-00589b99be)
Feb  9 03:50:46 controller proxy-server: STDERR: ERROR:root:Error limiting server controller:11211 (txn: txf905851da78b4ca9be972-00589b99be)
Feb  9 03:50:47 controller proxy-server: Account HEAD returning 503 for [] (txn: txf905851da78b4ca9be972-00589b99be) (client_ip: 192.168.1.191)
Feb  9 03:50:47 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/47 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - txf905851da78b4ca9be972-00589b99be - 0.2442 - - 1486592446.899775028 1486592447.143946886 -
Feb  9 03:50:47 controller swift: Account HEAD returning 503 for [] (txn: txb2fa7b4f670d4860b6acc-00589b99bf)
Feb  9 03:50:47 controller swift: - - 08/Feb/2017/22/20/47 GET /v1/.misplaced_objects/1486591200%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txb2fa7b4f670d4860b6acc-00589b99bf - 0.0085 - - 1486592447.826589108 1486592447.835084915 -
Feb  9 03:50:47 controller swift: Account GET returning 503 for [] (txn: tx0979b5d107d14435bed4b-00589b99bf)
Feb  9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0)
Feb  9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0)
Feb  9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0)
Feb  9 03:50:48 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: tx35d8c441a5fe4cefb1eda-00589b99c0)
Feb  9 03:50:48 controller swift: Account HEAD returning 503 for [507, 507, 507, 507] (txn: tx35d8c441a5fe4cefb1eda-00589b99c0)
Feb  9 03:50:49 controller swift: Account GET returning 503 for [] (txn: txd8343cd08e9840eeb59df-00589b99c1)
Feb  9 03:50:49 controller swift: - - 08/Feb/2017/22/20/49 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx0979b5d107d14435bed4b-00589b99bf - 2.0123 - - 1486592447.836558104 1486592449.848830938 - (txn: txd8343cd08e9840eeb59df-00589b99c1)
Feb  9 03:50:50 controller swift: Account HEAD returning 503 for [] (txn: txa62279ce22c5442c86935-00589b99c2)
Feb  9 03:50:51 controller proxy-server: Deferring reject downstream
Feb  9 03:50:51 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0e68587b841145abb2457-00589b99c3)
Feb  9 03:50:51 controller proxy-server: - - 08/Feb/2017/22/20/51 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx0e68587b841145abb2457-00589b99c3 - 0.0031 RL - 1486592451.151578903 1486592451.154645920 -
Feb  9 03:50:51 controller proxy-server: Account HEAD returning 503 for [] (txn: tx0e68587b841145abb2457-00589b99c3) (client_ip: 192.168.1.191)
Feb  9 03:50:51 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/51 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx0e68587b841145abb2457-00589b99c3 - 0.2539 - - 1486592451.149534941 1486592451.403433084 -
Feb  9 03:50:53 controller swift: Account GET returning 503 for [] (txn: tx59094c241e584f848c732-00589b99c5)
Feb  9 03:50:53 controller swift: - - 08/Feb/2017/22/20/53 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - txd8343cd08e9840eeb59df-00589b99c1 - 4.0157 - - 1486592449.843744040 1486592453.859456062 - (txn: tx59094c241e584f848c732-00589b99c5)
Feb  9 03:50:53 controller container-reconciler: Error listing containers in account .misplaced_objects (Unexpected response: 503 Service Unavailable) (txn: tx59094c241e584f848c732-00589b99c5)
Feb  9 03:50:53 controller swift: - - 08/Feb/2017/22/20/53 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx59094c241e584f848c732-00589b99c5 - 0.0072 - - 1486592453.853568077 1486592453.860737085 -
Feb  9 03:50:53 controller container-reconciler: Reconciler Stats: {} (txn: tx59094c241e584f848c732-00589b99c5)
Feb  9 03:50:53 controller container-reconciler: sleeping between intervals (30s) (txn: tx59094c241e584f848c732-00589b99c5)
Feb  9 03:50:54 controller swift: Account HEAD returning 503 for [] (txn: tx41ce67418b754a8ea9f22-00589b99c6)
Feb  9 03:50:54 controller object-expirer: Unhandled exception: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 172, in run_once#012    self.swift.get_account_info(self.expiring_objects_account)#012  File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 361, in get_account_info#012    resp = self.make_request('HEAD', path, {}, acceptable_statuses)#012  File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 195, in make_request#012    _('Unexpected response: %s') % resp.status, resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn: tx41ce67418b754a8ea9f22-00589b99c6)
Feb  9 03:50:59 controller proxy-server: Deferring reject downstream
Feb  9 03:50:59 controller proxy-server: Account HEAD returning 503 for [] (txn: tx4ae4184315c74bd6acc70-00589b99cb)
Feb  9 03:50:59 controller proxy-server: - - 08/Feb/2017/22/20/59 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx4ae4184315c74bd6acc70-00589b99cb - 0.0028 RL - 1486592459.413984060 1486592459.416832924 -
Feb  9 03:50:59 controller proxy-server: Account HEAD returning 503 for [] (txn: tx4ae4184315c74bd6acc70-00589b99cb) (client_ip: 192.168.1.191)
Feb  9 03:50:59 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/20/59 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx4ae4184315c74bd6acc70-00589b99cb - 0.2364 - - 1486592459.411684990 1486592459.648108006 -
Feb  9 03:51:15 controller proxy-server: Deferring reject downstream
Feb  9 03:51:15 controller proxy-server: Account HEAD returning 503 for [] (txn: tx635b33ff8b4545afae451-00589b99db)
Feb  9 03:51:15 controller proxy-server: - - 08/Feb/2017/22/21/15 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - Swift - - - - tx635b33ff8b4545afae451-00589b99db - 0.0027 RL - 1486592475.671029091 1486592475.673742056 -
Feb  9 03:51:15 controller proxy-server: Account HEAD returning 503 for [] (txn: tx635b33ff8b4545afae451-00589b99db) (client_ip: 192.168.1.191)
Feb  9 03:51:15 controller proxy-server: 192.168.1.191 192.168.1.191 08/Feb/2017/22/21/15 HEAD /v1/AUTH_3babe56d610847c389f89c959f74dd44 HTTP/1.0 503 - python-swiftclient-3.1.0 gAAAAABYm5m7i93P... - - - tx635b33ff8b4545afae451-00589b99db - 0.2344 - - 1486592475.669040918 1486592475.903404951 -
Feb  9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdb1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3)
Feb  9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdc1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3)
Feb  9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.194:6002/sdc1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3)
Feb  9 03:51:23 controller swift: ERROR Insufficient Storage 192.168.1.195:6002/sdb1 (txn: txe4e9a293b2be4d9ea5f62-00589b99e3)
Feb  9 03:51:23 controller swift: Account HEAD returning 503 for [507, 507, 507, 507] (txn: txe4e9a293b2be4d9ea5f62-00589b99e3)
Feb  9 03:51:23 controller swift: - - 08/Feb/2017/22/21/23 GET /v1/.misplaced_objects/1486591200%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txe4e9a293b2be4d9ea5f62-00589b99e3 - 0.0346 - - 1486592483.881172895 1486592483.915751934 -
Feb  9 03:51:23 controller swift: Account GET returning 503 for [] (txn: tx3a6089ddf65842b2ad7a1-00589b99e3)
Feb  9 03:51:25 controller swift: Account GET returning 503 for [] (txn: tx85f3f706856748e9a8b58-00589b99e5)
Feb  9 03:51:25 controller swift: - - 08/Feb/2017/22/21/25 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx3a6089ddf65842b2ad7a1-00589b99e3 - 2.0101 - - 1486592483.916512966 1486592485.926599979 - (txn: tx85f3f706856748e9a8b58-00589b99e5)
Feb  9 03:51:29 controller swift: Account GET returning 503 for [] (txn: tx27733d4774f940879eb3b-00589b99e9)
Feb  9 03:51:29 controller swift: - - 08/Feb/2017/22/21/29 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx85f3f706856748e9a8b58-00589b99e5 - 4.0149 - - 1486592485.920792103 1486592489.935691118 - (txn: tx27733d4774f940879eb3b-00589b99e9)
Feb  9 03:51:29 controller container-reconciler: Error listing containers in account .misplaced_objects (Unexpected response: 503 Service Unavailable) (txn: tx27733d4774f940879eb3b-00589b99e9)
Feb  9 03:51:29 controller swift: - - 08/Feb/2017/22/21/29 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 499 - Swift%20Container%20Reconciler - - 118 - tx27733d4774f940879eb3b-00589b99e9 - 0.0062 - - 1486592489.930593967 1486592489.936772108 -
Feb  9 03:51:29 controller container-reconciler: Reconciler Stats: {} (txn: tx27733d4774f940879eb3b-00589b99e9)
Feb  9 03:51:29 controller container-reconciler: sleeping between intervals (30s) (txn: tx27733d4774f940879eb3b-00589b99e9)

tx27733d4774f940879eb3b-00589b99e9)