Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG: PORT="11211" USER="memcached" MAXCONN="1024" CACHESIZE="64" OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG: CONFIG:

PORT="11211"
 USER="memcached"
 MAXCONN="1024"
 CACHESIZE="64"
 OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

valid?

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

OK, that log has been solved..

memcached service has to be launched before swift services. Ok, now... When trying "swift stat":

Sep 24 20:33:03 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: Account HEAD returning 503 for [] (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)

At the end... Connection refused.

If I netstat by 11211 port on Controller node (where memcached is suposed to be listening) i get:

[root@zeus admin]# netstat -putona | grep 11211
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2100/memcached      off (0.00/0/0)
tcp        0      0 192.168.10.2:46290          192.168.10.2:11211          ESTABLISHED 2739/python         off (0.00/0/0)
tcp        0      0 192.168.10.2:11211          192.168.10.2:46290          ESTABLISHED 2100/memcached      keepalive (6964,61/0/0)
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2100/memcached      off (0.00/0/0)

So I guess is listening... Then... why controller refuses connection?

Problem verifying installation of Swift.

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

OK, that log has been solved..

memcached service has to be launched before swift services. Ok, now... When trying "swift stat":

Sep 24 20:33:03 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: Account HEAD returning 503 for [] (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)

At the end... Connection refused.

If I netstat by 11211 port on Controller node (where memcached is suposed to be listening) i get:

[root@zeus admin]# netstat -putona | grep 11211
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2100/memcached      off (0.00/0/0)
tcp        0      0 192.168.10.2:46290          192.168.10.2:11211          ESTABLISHED 2739/python         off (0.00/0/0)
tcp        0      0 192.168.10.2:11211          192.168.10.2:46290          ESTABLISHED 2100/memcached      keepalive (6964,61/0/0)
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2100/memcached      off (0.00/0/0)

So I guess is listening... Then... why controller refuses connection?

EDITED:

Ok, fixed errors on /var/log/messages by adding and changing bind_ip with storage network IP in proxy-server.conf (on controller) and container, account and object conf files on Storage nodes.

But, still same issue when swift stat (connecion refused).

[root@zeus admin]# swift stat 
[Errno 111] Connection refused

Now, I don't know how to track errors. Any idea?

Problem verifying installation of Swift.Swift. [Errno 111] Connection refused

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

OK, that log has been solved..

memcached service has to be launched before swift services. Ok, now... When trying "swift stat":

Sep 24 20:33:03 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: Account HEAD returning 503 for [] (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)

At the end... Connection refused.

If I netstat by 11211 port on Controller node (where memcached is suposed to be listening) i get:

[root@zeus admin]# netstat -putona | grep 11211
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2100/memcached      off (0.00/0/0)
tcp        0      0 192.168.10.2:46290          192.168.10.2:11211          ESTABLISHED 2739/python         off (0.00/0/0)
tcp        0      0 192.168.10.2:11211          192.168.10.2:46290          ESTABLISHED 2100/memcached      keepalive (6964,61/0/0)
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2100/memcached      off (0.00/0/0)

So I guess is listening... Then... why controller refuses connection?

EDITED:

Ok, fixed errors on /var/log/messages by adding and changing bind_ip with storage network IP in proxy-server.conf (on controller) and container, account and object conf files on Storage nodes.

But, still same issue when swift stat (connecion refused).

[root@zeus admin]# swift stat 
[Errno 111] Connection refused

Now, I don't know how to track errors. Any idea?

Problem verifying installation of Swift. [Errno 111] Connection refused

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

OK, that log has been solved..

memcached service has to be launched before swift services. Ok, now... When trying "swift stat":

Sep 24 20:33:03 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: Account HEAD returning 503 for [] (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)

At the end... Connection refused.

If I netstat by 11211 port on Controller node (where memcached is suposed to be listening) i get:

[root@zeus admin]# netstat -putona | grep 11211
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2100/memcached      off (0.00/0/0)
tcp        0      0 192.168.10.2:46290          192.168.10.2:11211          ESTABLISHED 2739/python         off (0.00/0/0)
tcp        0      0 192.168.10.2:11211          192.168.10.2:46290          ESTABLISHED 2100/memcached      keepalive (6964,61/0/0)
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2100/memcached      off (0.00/0/0)

So I guess is listening... Then... why controller refuses connection?

EDITED:

Ok, fixed errors on /var/log/messages by adding and changing bind_ip with storage network IP in proxy-server.conf (on controller) and container, account and object conf files on Storage nodes.

But, still same issue when swift stat (connecion refused).

[root@zeus admin]# swift stat 
[Errno 111] Connection refused

Now, I don't know how to track errors. Any idea?

EDITED ON REPLY TO guimaluf:

If the connections is been refused, probably means your rsyncd is not properly configured.

Is rsyncd running? Paste rsyncd.conf and check if ports and IP are correct set.

RSYNCD.conf on Storage Node 1:

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.20.3 # storage network IP
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

RSYNCD.conf on Storage Node 2:

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.20.4
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

Which ports are open? Run netstat -lvpn and check it out.

NETSTAT ON PROXY NODE

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2102/memcached      
tcp        0      0 0.0.0.0:9292                0.0.0.0:*                   LISTEN      2535/python         
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1799/rpcbind        
tcp        0      0 192.168.20.2:8080           0.0.0.0:*                   LISTEN      4936/python         
tcp        0      0 127.0.0.1:5939              0.0.0.0:*                   LISTEN      1963/teamviewerd    
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2119/sshd           
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1964/cupsd          
tcp        0      0 0.0.0.0:48056               0.0.0.0:*                   LISTEN      1936/rpc.statd      
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2384/master         
tcp        0      0 0.0.0.0:35357               0.0.0.0:*                   LISTEN      5021/python         
tcp        0      0 0.0.0.0:6080                0.0.0.0:*                   LISTEN      2639/python         
tcp        0      0 0.0.0.0:9696                0.0.0.0:*                   LISTEN      2508/python         
tcp        0      0 0.0.0.0:8773                0.0.0.0:*                   LISTEN      2607/python         
tcp        0      0 0.0.0.0:8774                0.0.0.0:*                   LISTEN      2607/python         
tcp        0      0 0.0.0.0:8775                0.0.0.0:*                   LISTEN      2607/python         
tcp        0      0 0.0.0.0:9191                0.0.0.0:*                   LISTEN      2576/python         
tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN      5021/python         
tcp        0      0 0.0.0.0:8776                0.0.0.0:*                   LISTEN      2516/python         
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2437/qpidd          
tcp        0      0 192.168.10.2:3306           0.0.0.0:*                   LISTEN      2288/mysqld         
tcp        0      0 :::111                      :::*                        LISTEN      1799/rpcbind        
tcp        0      0 :::80                       :::*                        LISTEN      2416/httpd          
tcp        0      0 :::22                       :::*                        LISTEN      2119/sshd           
tcp        0      0 ::1:631                     :::*                        LISTEN      1964/cupsd          
tcp        0      0 ::1:25                      :::*                        LISTEN      2384/master         
tcp        0      0 :::33026                    :::*                        LISTEN      1936/rpc.statd      
tcp        0      0 :::5672                     :::*                        LISTEN      2437/qpidd          
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1799/rpcbind        
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1964/cupsd          
udp        0      0 192.168.20.2:123            0.0.0.0:*                               2127/ntpd           
udp        0      0 192.168.10.2:123            0.0.0.0:*                               2127/ntpd           
udp        0      0 10.51.1.116:123             0.0.0.0:*                               2127/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2127/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2127/ntpd           
udp        0      0 0.0.0.0:702                 0.0.0.0:*                               1799/rpcbind        
udp        0      0 0.0.0.0:53440               0.0.0.0:*                               1936/rpc.statd      
udp        0      0 0.0.0.0:840                 0.0.0.0:*                               1936/rpc.statd      
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2102/memcached      
udp        0      0 :::111                      :::*                                    1799/rpcbind        
udp        0      0 fe80::20e:cff:fe6b:d84a:123 :::*                                    2127/ntpd           
udp        0      0 fe80::219:d1ff:fe8f:fc54:123 :::*                                    2127/ntpd           
udp        0      0 ::1:123                     :::*                                    2127/ntpd           
udp        0      0 :::123                      :::*                                    2127/ntpd           
udp        0      0 :::702                      :::*                                    1799/rpcbind        
udp        0      0 :::43488                    :::*                                    1936/rpc.statd      
netstat: no support for `AF INET (sctp)' on this system.
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     13422  1999/hald           @/var/run/hald/dbus-iZZEOzrwgR
unix  2      [ ACC ]     STREAM     LISTENING     23747  3248/gnome-panel    /tmp/orbit-admin/linc-cb0-0-58b876da5c3ef
unix  2      [ ACC ]     STREAM     LISTENING     24117  3288/bonobo-activat /tmp/orbit-admin/linc-cd8-0-58a5d3b9a6ee9
unix  2      [ ACC ]     STREAM     LISTENING     24291  3314/gconf-helper   /tmp/orbit-admin/linc-cf2-0-5bd3f8c8b738d
unix  2      [ ACC ]     STREAM     LISTENING     24344  3310/trashapplet    /tmp/orbit-admin/linc-cee-0-3becc4c5e9047
unix  2      [ ACC ]     STREAM     LISTENING     24348  3311/wnck-applet    /tmp/orbit-admin/linc-cef-0-17c90e97e91c2
unix  2      [ ACC ]     STREAM     LISTENING     13378  1990/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     25067  3375/nm-applet      /tmp/orbit-admin/linc-d2f-0-1fe57b670c33
unix  2      [ ACC ]     STREAM     LISTENING     25115  3437/gnome-screensa /tmp/orbit-admin/linc-d49-0-28275bb8b8d95
unix  2      [ ACC ]     STREAM     LISTENING     25326  3417/gpk-update-ico /tmp/orbit-admin/linc-d59-0-7e09410ed4fca
unix  2      [ ACC ]     STREAM     LISTENING     25353  3346/evolution-alar /tmp/orbit-admin/linc-d12-0-7ed02077657c
unix  2      [ ACC ]     STREAM     LISTENING     25478  3357/gnome-power-ma /tmp/orbit-admin/linc-d1d-0-6ed384796b800
unix  2      [ ACC ]     STREAM     LISTENING     25514  3279/nautilus       /tmp/orbit-admin/linc-ccf-0-1dd0bb857d0e5
unix  2      [ ACC ]     STREAM     LISTENING     25748  3445/gdm-user-switc /tmp/orbit-admin/linc-d75-0-2e32c8844a7bd
unix  2      [ ACC ]     STREAM     LISTENING     25769  3448/notification-a /tmp/orbit-admin/linc-d78-0-611c45154d3e5
unix  2      [ ACC ]     STREAM     LISTENING     25789  3449/clock-applet   /tmp/orbit-admin/linc-d79-0-59f173a853607
unix  2      [ ACC ]     STREAM     LISTENING     25999  3616/gconf-im-setti /tmp/orbit-admin/linc-e20-0-7b66f0f8de66d
unix  2      [ ACC ]     STREAM     LISTENING     26039  3440/gnote          /tmp/orbit-admin/linc-d70-0-48b152a6cb167
unix  2      [ ACC ]     STREAM     LISTENING     26147  3639/wineserver     socket
unix  2      [ ACC ]     STREAM     LISTENING     24238  3247/pulseaudio     /tmp/.esd-500/socket
unix  2      [ ACC ]     STREAM     LISTENING     24242  3247/pulseaudio     /home/admin/.pulse/705c70fcec307615a8c014ef00000038-runtime/native
unix  2      [ ACC ]     STREAM     LISTENING     8867   1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     558953 10451/gedit         /tmp/gedit.admin.623590353
unix  2      [ ACC ]     STREAM     LISTENING     14962  2288/mysqld         /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     17234  2741/Xorg           @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     15186  2384/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     15193  2384/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     15197  2384/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     15201  2384/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     15205  2384/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     15357  2408/abrtd          /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     348033 5126/firefox        /tmp/orbit-admin/linc-1406-0-78ca38b34445
unix  2      [ ACC ]     STREAM     LISTENING     22839  3137/gnome-session  @/tmp/.ICE-unix/3137
unix  2      [ ACC ]     STREAM     LISTENING     558961 10451/gedit         /tmp/orbit-admin/linc-28d3-0-12ac38b78759d
unix  2      [ ACC ]     STREAM     LISTENING     15209  2384/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     15213  2384/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     17235  2741/Xorg           /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     22840  3137/gnome-session  /tmp/.ICE-unix/3137
unix  2      [ ACC ]     STREAM     LISTENING     17399  2738/gdm-simple-sla @/tmp/gdm-greeter-INYMRsSV
unix  2      [ ACC ]     STREAM     LISTENING     22865  3220/gconfd-2       /tmp/orbit-admin/linc-c94-0-5ea0540a3bdd8
unix  2      [ ACC ]     STREAM     LISTENING     17584  2738/gdm-simple-sla @/tmp/gdm-session-TlcsJVrd
unix  2      [ ACC ]     STREAM     LISTENING     13296  1964/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     23119  3137/gnome-session  /tmp/orbit-admin/linc-c41-0-6760a38d472e6
unix  2      [ ACC ]     STREAM     LISTENING     15423  2416/httpd          /etc/httpd/run/wsgi.2416.0.1.sock
unix  2      [ ACC ]     STREAM     LISTENING     15217  2384/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     15221  2384/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     15225  2384/master         private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     15229  2384/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     23328  3127/gnome-keyring- /tmp/orbit-admin/linc-c37-0-2384310c9a33e
unix  2      [ ACC ]     STREAM     LISTENING     12888  1799/rpcbind        /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     23334  3127/gnome-keyring- /tmp/keyring-fZ8CMy/socket.ssh
unix  2      [ ACC ]     STREAM     LISTENING     23343  3127/gnome-keyring- /tmp/keyring-fZ8CMy/socket.pkcs11
unix  2      [ ACC ]     STREAM     LISTENING     23344  3228/gnome-settings /tmp/orbit-admin/linc-c9c-0-624ae00a9b096
unix  2      [ ACC ]     STREAM     LISTENING     15233  2384/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     15237  2384/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     13070  1917/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     24750  3353/bluetooth-appl /tmp/orbit-admin/linc-d19-0-61cf19df820bd
unix  2      [ ACC ]     STREAM     LISTENING     15241  2384/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     22681  3146/dbus-daemon    @/tmp/dbus-LVuDSvuzZX
unix  2      [ ACC ]     STREAM     LISTENING     15245  2384/master         private/retry
unix  2      [ ACC ]     STREAM     LISTENING     31434  3724/gnome-terminal /tmp/orbit-admin/linc-e8c-0-6d27754be7f85
unix  2      [ ACC ]     STREAM     LISTENING     13415  1999/hald           @/var/run/hald/dbus-IYPDQvbH8x
unix  2      [ ACC ]     STREAM     LISTENING     22363  3127/gnome-keyring- /tmp/keyring-fZ8CMy/socket
unix  2      [ ACC ]     STREAM     LISTENING     23560  3243/metacity       /tmp/orbit-admin/linc-cab-0-786355d72e1f9
unix  2      [ ACC ]     STREAM     LISTENING     23611  3230/seahorse-daemo /tmp/orbit-admin/linc-c9e-0-60a259b333ae6
unix  2      [ ACC ]     STREAM     LISTENING     15249  2384/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     15253  2384/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     15257  2384/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     15261  2384/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     15265  2384/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     15269  2384/master         private/scache
netstat: no support for `AF IPX' on this system.
netstat: no support for `AF AX25' on this system.
netstat: no support for `AF X25' on this system.
netstat: no support for `AF NETROM' on this system.

NETSTAT ON STORAGE NODE 1

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      2036/rpcbind        
tcp        0      0 192.168.20.3:6000           0.0.0.0:*                   LISTEN      14360/python        
tcp        0      0 192.168.20.3:6001           0.0.0.0:*                   LISTEN      14354/python        
tcp        0      0 192.168.20.3:6002           0.0.0.0:*                   LISTEN      14355/python        
tcp        0      0 127.0.0.1:5939              0.0.0.0:*                   LISTEN      1902/teamviewerd    
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2345/sshd           
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      2194/cupsd          
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2441/master         
tcp        0      0 0.0.0.0:3260                0.0.0.0:*                   LISTEN      2320/tgtd           
tcp        0      0 0.0.0.0:59517               0.0.0.0:*                   LISTEN      2167/rpc.statd      
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2476/qpidd          
tcp        0      0 :::111                      :::*                        LISTEN      2036/rpcbind        
tcp        0      0 :::22                       :::*                        LISTEN      2345/sshd           
tcp        0      0 ::1:631                     :::*                        LISTEN      2194/cupsd          
tcp        0      0 ::1:25                      :::*                        LISTEN      2441/master         
tcp        0      0 :::3260                     :::*                        LISTEN      2320/tgtd           
tcp        0      0 :::5672                     :::*                        LISTEN      2476/qpidd          
tcp        0      0 :::873                      :::*                        LISTEN      2353/xinetd         
tcp        0      0 :::37385                    :::*                        LISTEN      2167/rpc.statd      
udp        0      0 0.0.0.0:58221               0.0.0.0:*                               2167/rpc.statd      
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               2036/rpcbind        
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               2194/cupsd          
udp        0      0 10.51.1.117:123             0.0.0.0:*                               2361/ntpd           
udp        0      0 192.168.20.3:123            0.0.0.0:*                               2361/ntpd           
udp        0      0 192.168.11.3:123            0.0.0.0:*                               2361/ntpd           
udp        0      0 192.168.10.3:123            0.0.0.0:*                               2361/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2361/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2361/ntpd           
udp        0      0 0.0.0.0:647                 0.0.0.0:*                               2167/rpc.statd      
udp        0      0 0.0.0.0:939                 0.0.0.0:*                               2036/rpcbind        
udp        0      0 :::111                      :::*                                    2036/rpcbind        
udp        0      0 fe80::f82c:21ff:fe51:2f6:123 :::*                                    2361/ntpd           
udp        0      0 fe80::21c:c0ff:fe06:7428:123 :::*                                    2361/ntpd           
udp        0      0 fe80::38f1:a6ff:fec8:22d:123 :::*                                    2361/ntpd           
udp        0      0 fe80::4010:82ff:fe2d:df8:123 :::*                                    2361/ntpd           
udp        0      0 fe80::248:54ff:fe4c:2440:123 :::*                                    2361/ntpd           
udp        0      0 ::1:123                     :::*                                    2361/ntpd           
udp        0      0 :::123                      :::*                                    2361/ntpd           
udp        0      0 :::939                      :::*                                    2036/rpcbind        
udp        0      0 :::48743                    :::*                                    2167/rpc.statd      
netstat: no support for `AF INET (sctp)' on this system.
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     22321  1459/ovs-vswitchd   /var/run/openvswitch/br-tun.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     19170  2751/Xorg           /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     22214  2937/gnome-session  /tmp/.ICE-unix/2937
unix  2      [ ACC ]     STREAM     LISTENING     17328  2441/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     23041  3088/gconfd-2       /tmp/orbit-gdm/linc-c10-0-5d28750446c53
unix  2      [ ACC ]     STREAM     LISTENING     17332  2441/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     23115  2937/gnome-session  /tmp/orbit-gdm/linc-b79-0-5ed1d1874eb3c
unix  2      [ ACC ]     STREAM     LISTENING     24246  3153/gnome-settings /tmp/orbit-gdm/linc-c51-0-54a27fe94f33d
unix  2      [ ACC ]     STREAM     LISTENING     24579  3161/bonobo-activat /tmp/orbit-gdm/linc-c59-0-6c508cf82554c
unix  2      [ ACC ]     STREAM     LISTENING     24585  3151/at-spi-registr /tmp/orbit-gdm/linc-c4f-0-7587a01325aec
unix  2      [ ACC ]     STREAM     LISTENING     26013  3287/metacity       /tmp/orbit-gdm/linc-cd7-0-528450cb67547
unix  2      [ ACC ]     STREAM     LISTENING     22323  1459/ovs-vswitchd   /var/run/openvswitch/br-tun.snoop
unix  2      [ ACC ]     STREAM     LISTENING     26839  3372/gnome-power-ma /tmp/orbit-gdm/linc-d2c-0-378b94a92404e
unix  2      [ ACC ]     STREAM     LISTENING     19269  2544/python         /var/lib/neutron/metadata_proxy
unix  2      [ ACC ]     STREAM     LISTENING     13993  1459/ovs-vswitchd   /var/run/openvswitch/br-ex.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     27081  3392/polkit-gnome-a /tmp/orbit-gdm/linc-d40-0-1911b5bfc2
unix  2      [ ACC ]     STREAM     LISTENING     27120  3418/gdm-simple-gre /tmp/orbit-gdm/linc-d5a-0-586cc91363e23
unix  2      [ ACC ]     STREAM     LISTENING     13995  1459/ovs-vswitchd   /var/run/openvswitch/br-ex.snoop
unix  2      [ ACC ]     STREAM     LISTENING     16215  2228/hald           @/var/run/hald/dbus-BiG0S6mixe
unix  2      [ ACC ]     STREAM     LISTENING     30075  3536/pulseaudio     /var/lib/gdm/.pulse/95aea038884b2057483add8800000039-runtime/native
unix  2      [ ACC ]     STREAM     LISTENING     16164  2219/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     8634   1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     16820  2320/tgtd           /var/run/tgtd.ipc_abstract_namespace.0
unix  2      [ ACC ]     STREAM     LISTENING     19169  2751/Xorg           @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     17336  2441/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     17340  2441/master         private/scache
unix  2      [ ACC ]     STREAM     LISTENING     17257  2441/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     17264  2441/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     20516  2720/gdm-simple-sla @/tmp/gdm-greeter-AeCcMcTu
unix  2      [ ACC ]     STREAM     LISTENING     17268  2441/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     21226  2931/dbus-daemon    @/tmp/dbus-CmjxSqH8lP
unix  2      [ ACC ]     STREAM     LISTENING     17272  2441/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     17276  2441/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     17280  2441/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     17284  2441/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     16086  2194/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     16208  2228/hald           @/var/run/hald/dbus-N9C6uB5ZGa
unix  2      [ ACC ]     STREAM     LISTENING     17288  2441/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     17292  2441/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     15552  2036/rpcbind        /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     17296  2441/master         private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     17300  2441/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     17429  2465/abrtd          /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     21762  2720/gdm-simple-sla @/tmp/gdm-session-NjSEEfOG
unix  2      [ ACC ]     STREAM     LISTENING     13609  1448/ovsdb-server   /var/run/openvswitch/db.sock
unix  2      [ ACC ]     STREAM     LISTENING     13611  1448/ovsdb-server   /var/run/openvswitch/ovsdb-server.1448.ctl
unix  2      [ ACC ]     STREAM     LISTENING     17304  2441/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     17308  2441/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     17312  2441/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     17316  2441/master         private/retry
unix  2      [ ACC ]     STREAM     LISTENING     22213  2937/gnome-session  @/tmp/.ICE-unix/2937
unix  2      [ ACC ]     STREAM     LISTENING     17320  2441/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     13651  1459/ovs-vswitchd   /var/run/openvswitch/ovs-vswitchd.1459.ctl
unix  2      [ ACC ]     STREAM     LISTENING     17324  2441/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     15749  2148/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     13689  1459/ovs-vswitchd   /var/run/openvswitch/br-int.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     13691  1459/ovs-vswitchd   /var/run/openvswitch/br-int.snoop

NETSTAT ON STORAGE NODE 2

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:59694               0.0.0.0:*                   LISTEN      2246/rpc.statd      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      2116/rpcbind        
tcp        0      0 192.168.20.4:6000           0.0.0.0:*                   LISTEN      15841/python        
tcp        0      0 192.168.20.4:6001           0.0.0.0:*                   LISTEN      15835/python        
tcp        0      0 192.168.20.4:6002           0.0.0.0:*                   LISTEN      15836/python        
tcp        0      0 127.0.0.1:5939              0.0.0.0:*                   LISTEN      1978/teamviewerd    
tcp        0      0 192.168.122.1:53            0.0.0.0:*                   LISTEN      2796/dnsmasq        
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2401/sshd           
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      2274/cupsd          
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2497/master         
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2547/qpidd          
tcp        0      0 :::36075                    :::*                        LISTEN      2246/rpc.statd      
tcp        0      0 :::111                      :::*                        LISTEN      2116/rpcbind        
tcp        0      0 :::22                       :::*                        LISTEN      2401/sshd           
tcp        0      0 ::1:631                     :::*                        LISTEN      2274/cupsd          
tcp        0      0 ::1:25                      :::*                        LISTEN      2497/master         
tcp        0      0 :::5672                     :::*                        LISTEN      2547/qpidd          
tcp        0      0 :::873                      :::*                        LISTEN      2409/xinetd         
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               2116/rpcbind        
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               2274/cupsd          
udp        0      0 192.168.122.1:123           0.0.0.0:*                               2417/ntpd           
udp        0      0 192.168.20.4:123            0.0.0.0:*                               2417/ntpd           
udp        0      0 192.168.11.4:123            0.0.0.0:*                               2417/ntpd           
udp        0      0 192.168.10.4:123            0.0.0.0:*                               2417/ntpd           
udp        0      0 10.51.1.118:123             0.0.0.0:*                               2417/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2417/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2417/ntpd           
udp        0      0 0.0.0.0:1019                0.0.0.0:*                               2116/rpcbind        
udp        0      0 192.168.122.1:53            0.0.0.0:*                               2796/dnsmasq        
udp        0      0 0.0.0.0:54072               0.0.0.0:*                               2246/rpc.statd      
udp        0      0 0.0.0.0:67                  0.0.0.0:*                               2796/dnsmasq        
udp        0      0 0.0.0.0:726                 0.0.0.0:*                               2246/rpc.statd      
udp        0      0 :::111                      :::*                                    2116/rpcbind        
udp        0      0 fe80::8c6d:e3ff:feb5:b97:123 :::*                                    2417/ntpd           
udp        0      0 fe80::3815:16ff:fe7b:47b:123 :::*                                    2417/ntpd           
udp        0      0 fe80::3c08:8aff:fe0b:435:123 :::*                                    2417/ntpd           
udp        0      0 fe80::a838:9cff:fe21:998:123 :::*                                    2417/ntpd           
udp        0      0 fe80::cc48:cfff:feca:d62:123 :::*                                    2417/ntpd           
udp        0      0 fe80::e07d:ffff:fe83:ad9:123 :::*                                    2417/ntpd           
udp        0      0 fe80::a0ad:d0ff:fe71:700:123 :::*                                    2417/ntpd           
udp        0      0 fe80::88ca:b1ff:fef2:966:123 :::*                                    2417/ntpd           
udp        0      0 ::1:123                     :::*                                    2417/ntpd           
udp        0      0 fe80::7271:bcff:fe94:97f:123 :::*                                    2417/ntpd           
udp        0      0 fe80::201:2ff:feb0:ebd3:123 :::*                                    2417/ntpd           
udp        0      0 :::123                      :::*                                    2417/ntpd           
udp        0      0 :::1019                     :::*                                    2116/rpcbind        
udp        0      0 :::60244                    :::*                                    2246/rpc.statd      
netstat: no support for `AF INET (sctp)' on this system.
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     20025  3103/dbus-daemon    @/tmp/dbus-0QyG3GyUun
unix  2      [ ACC ]     STREAM     LISTENING     16959  2497/master         private/scache
unix  2      [ ACC ]     STREAM     LISTENING     16026  2299/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     19231  2910/Xorg           /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     16070  2308/hald           @/var/run/hald/dbus-mm8mSFIP04
unix  2      [ ACC ]     STREAM     LISTENING     20102  3105/gnome-session  /tmp/.ICE-unix/3105
unix  2      [ ACC ]     STREAM     LISTENING     20169  3112/gconfd-2       /tmp/orbit-gdm/linc-c28-0-1e5e07d9934f7
unix  2      [ ACC ]     STREAM     LISTENING     14388  1629/ovs-vswitchd   /var/run/openvswitch/ovs-vswitchd.1629.ctl
unix  2      [ ACC ]     STREAM     LISTENING     20244  3105/gnome-session  /tmp/orbit-gdm/linc-c21-0-6bc2794398d3f
unix  2      [ ACC ]     STREAM     LISTENING     20342  3131/gnome-settings /tmp/orbit-gdm/linc-c3b-0-753fe0e8dbfbc
unix  2      [ ACC ]     STREAM     LISTENING     20426  3133/bonobo-activat /tmp/orbit-gdm/linc-c3d-0-4d6994157b8a2
unix  2      [ ACC ]     STREAM     LISTENING     20432  3126/at-spi-registr /tmp/orbit-gdm/linc-c36-0-539548d07ba6d
unix  2      [ ACC ]     STREAM     LISTENING     20510  3141/metacity       /tmp/orbit-gdm/linc-c45-0-19bf99e0354cc
unix  2      [ ACC ]     STREAM     LISTENING     21578  3143/gdm-simple-gre /tmp/orbit-gdm/linc-c47-0-72c89a3d6ca8c
unix  2      [ ACC ]     STREAM     LISTENING     21679  3146/polkit-gnome-a /tmp/orbit-gdm/linc-c4a-0-5997457a8f913
unix  2      [ ACC ]     STREAM     LISTENING     21748  3144/gnome-power-ma /tmp/orbit-gdm/linc-c48-0-1ce089c0b26d8
unix  2      [ ACC ]     STREAM     LISTENING     8818   1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     19890  2904/gdm-simple-sla @/tmp/gdm-greeter-pxNMOhtW
unix  2      [ ACC ]     STREAM     LISTENING     20067  2904/gdm-simple-sla @/tmp/gdm-session-lduoMDHu
unix  2      [ ACC ]     STREAM     LISTENING     19230  2910/Xorg           @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     17663  2607/libvirtd       /var/run/libvirt/libvirt-sock
unix  2      [ ACC ]     STREAM     LISTENING     17665  2607/libvirtd       /var/run/libvirt/libvirt-sock-ro
unix  2      [ ACC ]     STREAM     LISTENING     17052  2521/abrtd          /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     16876  2497/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     14419  1629/ovs-vswitchd   /var/run/openvswitch/br-int.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     16883  2497/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     16887  2497/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     16891  2497/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     14421  1629/ovs-vswitchd   /var/run/openvswitch/br-int.snoop
unix  2      [ ACC ]     STREAM     LISTENING     14346  1618/ovsdb-server   /var/run/openvswitch/db.sock
unix  2      [ ACC ]     STREAM     LISTENING     14348  1618/ovsdb-server   /var/run/openvswitch/ovsdb-server.1618.ctl
unix  2      [ ACC ]     STREAM     LISTENING     22733  1629/ovs-vswitchd   /var/run/openvswitch/br-tun.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     22735  1629/ovs-vswitchd   /var/run/openvswitch/br-tun.snoop
unix  2      [ ACC ]     STREAM     LISTENING     20101  3105/gnome-session  @/tmp/.ICE-unix/3105
unix  2      [ ACC ]     STREAM     LISTENING     16895  2497/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     16899  2497/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     16903  2497/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     16907  2497/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     15948  2274/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     16911  2497/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     16915  2497/master         private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     22163  3159/pulseaudio     /var/lib/gdm/.pulse/25fbee1e357459864436a51500000037-runtime/native
unix  2      [ ACC ]     STREAM     LISTENING     15520  2116/rpcbind        /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     16919  2497/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     16923  2497/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     16927  2497/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     16931  2497/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     16063  2308/hald           @/var/run/hald/dbus-JjnLasFYOG
unix  2      [ ACC ]     STREAM     LISTENING     15724  2227/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     16935  2497/master         private/retry
unix  2      [ ACC ]     STREAM     LISTENING     16939  2497/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     16943  2497/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     16947  2497/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     16951  2497/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     16955  2497/master         private/anvil

There is any firewall running? Is it blocking ports 873, 6000, 6001, 6002?

Nope, Iptables is disabled for testing first... :S

========

Ive just realize your problem it's happening also on proxy-node.

What endpoints are set on keystone? Also run swift --debug stat and paste for us

Ok, endpoints:

[root@zeus admin]# keystone endpoint-list
+----------------------------------+-----------+----------------------------------------+----------------------------------------+-----------------------------------+----------------------------------+
|                id                |   region  |               publicurl                |              internalurl               |              adminurl             |            service_id            |
+----------------------------------+-----------+----------------------------------------+----------------------------------------+-----------------------------------+----------------------------------+
| 4a0847cf7f9c4dc8a73bf76b63fa6a4f | regionOne |   http://zeus:8774/v2/%(tenant_id)s    |   http://zeus:8774/v2/%(tenant_id)s    | http://zeus:8774/v2/%(tenant_id)s | 7863d4a9dce648129315239e41c6a1ea |
| 8d84a88672f946828e4bd971b4215d61 | regionOne |         http://zeus:5000/v2.0          |         http://zeus:5000/v2.0          |       http://zeus:35357/v2.0      | 959fa3a77d8745a98c93f3e3a232b182 |
| 980ec695bc6942c8b866d7f04d125d26 | regionOne |            http://zeus:9696            |            http://zeus:9696            |          http://zeus:9696         | a30a90c5d4c140038d67c5144cf01214 |
| a9b5a188c79546e89beecb6a13841dd3 | regionOne | http://zeus:8080/v1/AUTH_%(tenant_id)s | http://zeus:8080/v1/AUTH_%(tenant_id)s |          http://zeus:8080         | a175bed646d647d2820728230155d9ce |
| bed17d64ed8c42b783ca7cf457cc63ed | regionOne |   http://zeus:8776/v1/%(tenant_id)s    |   http://zeus:8776/v1/%(tenant_id)s    | http://zeus:8776/v1/%(tenant_id)s | 60d60cc313a24f829d2cc24590864d78 |
| d97540ab45aa4ba299d4a7b228ca97e4 | regionOne |            http://zeus:9292            |            http://zeus:9292            |          http://zeus:9292         | fd09daf22ef145199795d2e118aca09c |
| e2b179f7370d45e0bf73aa7e39c9b8d5 | regionOne |   http://zeus:8776/v2/%(tenant_id)s    |   http://zeus:8776/v2/%(tenant_id)s    | http://zeus:8776/v2/%(tenant_id)s | 0d7a707e91ff44439bdaedae32f7b1d2 |
+----------------------------------+-----------+----------------------------------------+----------------------------------------+-----------------------------------+----------------------------------+

Now, swift --debug stat log:

DEBUG:keystoneclient.session:REQ: curl -i -X POST http://zeus:35357/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "passadmin"}}}'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 6819
DEBUG:keystoneclient.session:RESP: [200] {'date': 'Thu, 25 Sep 2014 12:36:27 GMT', 'content-type': 'application/json', 'content-length': '6819', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-09-25T12:36:27.032625", "expires": "2014-09-25T13:36:27Z", "id": "MIIMKwYJKoZIhvcNAQcCoIIMHDCCDBgCAQExCTAHBgUrDgMCGjCCCoEGCSqGSIb3DQEHAaCCCnIEggpueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wOS0yNVQxMjozNjoyNy4wMzI2MjUiLCAiZXhwaXJlcyI6ICIyMDE0LTA5LTI1VDEzOjM2OjI3WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIkFkbWluIFRlbmFudCIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgIm5hbWUiOiAiYWRtaW4ifX0sICJzZXJ2aWNlQ2F0YWxvZyI6IFt7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly96ZXVzOjg3NzQvdjIvMzdkOGIxZmI2MjM3NGVmOWE1NjQyZjk5MmY2MjRlZWMiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vemV1czo4Nzc0L3YyLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgImlkIjogIjM1NDNhNTBmNWNkZDQyNTVhZjg3NjUxODhmNjdlMTI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vemV1czo4Nzc0L3YyLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovL3pldXM6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly96ZXVzOjk2OTYiLCAiaWQiOiAiYTg2OWJmNzE2ZDdiNDU0OTllNjNhMTg1OTQ3ZDZhMjMiLCAicHVibGljVVJMIjogImh0dHA6Ly96ZXVzOjk2OTYifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAibmV0d29yayIsICJuYW1lIjogIm5ldXRyb24ifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czo4Nzc2L3YyLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92Mi8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyIsICJpZCI6ICI1Yzg3NjM5YmQ5YmU0MTNjODA3M2ZkOTYyZGY5NjIwMiIsICJwdWJsaWNVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92Mi8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWV2MiIsICJuYW1lIjogImNpbmRlcnYyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovL3pldXM6OTI5MiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly96ZXVzOjkyOTIiLCAiaWQiOiAiMDgwYmVjMmUwZmRmNGZmMzgyMGQ4Zjc5MjM5MWQ0NTUiLCAicHVibGljVVJMIjogImh0dHA6Ly96ZXVzOjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czo4Nzc2L3YxLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92MS8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyIsICJpZCI6ICI4YTJlYjg2NGIwYmY0ODkyYTBhYTIwOTQxZjhjMWJiZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92MS8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czo4MDgwIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6ODA4MC92MS9BVVRIXzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgImlkIjogIjMzNDk5YzczNTU5ZTRlZDlhMTZjYjNhY2Q3MmRjOGYzIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vemV1czo4MDgwL3YxL0FVVEhfMzdkOGIxZmI2MjM3NGVmOWE1NjQyZjk5MmY2MjRlZWMifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAib2JqZWN0LXN0b3JlIiwgIm5hbWUiOiAic3dpZnQifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6NTAwMC92Mi4wIiwgImlkIjogIjAwZTc3NTAxNWY3OTRiZDliZGUzNThiMTc2MDFhMmQyIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vemV1czo1MDAwL3YyLjAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaWRlbnRpdHkiLCAibmFtZSI6ICJrZXlzdG9uZSJ9XSwgInVzZXIiOiB7InVzZXJuYW1lIjogImFkbWluIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICI2NzJmOWZjMzAzZDQ0Yjk1OTE5NjczYjE0OTIzZWUwMCIsICJyb2xlcyI6IFt7Im5hbWUiOiAiX21lbWJlcl8ifSwgeyJuYW1lIjogImFkbWluIn1dLCAibmFtZSI6ICJhZG1pbiJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5ZmUyZmY5ZWU0Mzg0YjE4OTRhOTA4NzhkM2U5MmJhYiIsICJiYWQ4NDc3MGNiMWY0NjIzYmMxNGVjOTBjYTk5N2QyYiJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHuVuancCsf3LP+RQMrGQ8y6VSrUXq+tfj-vFhKeWs2Admm0aAnWt4yeusw6V4svw2QLkxh6ZG582yevtW02T4tDF86zVIPrcn2xYuhNhp0m6FC3n-Cbhd-LVmUDfT4RaFqitU1mjQ8Lgqycc+umgKJPOW9--k63gxdz1TlFufoBTM5xAPwGXK47WjQ9cRpw-CAYXpOwQEVV5iOZDOJzmx6J5B2+dxoJTImQXdoUMeeCadvcuwCo8bO5iDPtxjerA3wJhjA6JFIwHhpUX04Z0j7MNAQlk16dgJc-mXlHmr5GsmjkVEuUKyxAuDrA2wwRoaNjO4D9O-8P+10FC9fJ0oE=", "tenant": {"description": "Admin Tenant", "enabled": true, "id": "37d8b1fb62374ef9a5642f992f624eec", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://zeus:8774/v2/37d8b1fb62374ef9a5642f992f624eec", "region": "regionOne", "internalURL": "http://zeus:8774/v2/37d8b1fb62374ef9a5642f992f624eec", "id": "3543a50f5cdd4255af8765188f67e128", "publicURL": "http://zeus:8774/v2/37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://zeus:9696", "region": "regionOne", "internalURL": "http://zeus:9696", "id": "a869bf716d7b45499e63a185947d6a23", "publicURL": "http://zeus:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://zeus:8776/v2/37d8b1fb62374ef9a5642f992f624eec", "region": "regionOne", "internalURL": "http://zeus:8776/v2/37d8b1fb62374ef9a5642f992f624eec", "id": "5c87639bd9be413c8073fd962df96202", "publicURL": "http://zeus:8776/v2/37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://zeus:9292", "region": "regionOne", "internalURL": "http://zeus:9292", "id": "080bec2e0fdf4ff3820d8f792391d455", "publicURL": "http://zeus:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://zeus:8776/v1/37d8b1fb62374ef9a5642f992f624eec", "region": "regionOne", "internalURL": "http://zeus:8776/v1/37d8b1fb62374ef9a5642f992f624eec", "id": "8a2eb864b0bf4892a0aa20941f8c1bbe", "publicURL": "http://zeus:8776/v1/37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://zeus:8080", "region": "regionOne", "internalURL": "http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec", "id": "33499c73559e4ed9a16cb3acd72dc8f3", "publicURL": "http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://zeus:35357/v2.0", "region": "regionOne", "internalURL": "http://zeus:5000/v2.0", "id": "00e775015f794bd9bde358b17601a2d2", "publicURL": "http://zeus:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "672f9fc303d44b95919673b14923ee00", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "bad84770cb1f4623bc14ec90ca997d2b"]}}}

DEBUG:iso8601.iso8601:Parsed 2014-09-25T13:36:27Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'13', 'daydash': u'25', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'27', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'09', 'day': None, 'minute': u'36'} with default timezone <iso8601.iso8601.Utc object at 0x222a9d0>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'09' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 9 for 'month' with default 9
DEBUG:iso8601.iso8601:Got u'25' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 25 for 'day' with default 25
DEBUG:iso8601.iso8601:Got u'13' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'36' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'27' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
ERROR:swiftclient:[Errno 111] Connection refused
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1208, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 485, in head_account
    conn.request(method, parsed.path, '', headers)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 213, in request
    files=files, **self.requests_args)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 181, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 206, in send
    raise ConnectionError(sockerr)
ConnectionError: [Errno 111] Connection refused
[Errno 111] Connection refused

[solved] Problem verifying installation of Swift. [Errno 111] Connection refused

Hi, I just installed Swift on a fresh installation of OpenStack.

I've got a problem when doing swift stat:

Account HEAD failed: http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec 503 Internal Server Error

I think is about proxy server not being able to authenticate with Storage nodes, but I'm not pretty sure...

I've been reviewing my proxy-server.conf and I don't see any error...

Do you have any idea guys?

My proxy-server.conf: [DEFAULT] bind_port = 8080 user = swift

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# cache directory for signing certificate
#   signing_dir = /opt/swift/keystone-signing
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = zeus
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = passswift


[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

Just say that zeus is controller node (where keystone is installed).

Thanks

PD: All devices on Storage nodes are mounted.

EDITED:

From rsyslog: /var/log/messages

Sep 23 22:34:23 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:23 zeus proxy-server: Account GET returning 503 for [] (txn: tx8ff1e9e5b53a4824a84d9-005421d94f) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:24 zeus proxy-server: Account GET returning 503 for [] (txn: txc144fee25d804d499ab74-005421d950) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:26 zeus proxy-server: Account GET returning 503 for [] (txn: tx800f935d89ca434eba514-005421d952) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:30 zeus proxy-server: Account GET returning 503 for [] (txn: tx8fe5c4f6613e47038b17d-005421d956) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:38 zeus proxy-server: Account GET returning 503 for [] (txn: tx2b981864e96045f6a479a-005421d95e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to GET /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)
Sep 23 22:34:54 zeus proxy-server: Account GET returning 503 for [] (txn: tx7d278694dec7481c871ab-005421d96e) (client_ip: 192.168.10.2)

EDIT:

Ok, now I've got another log on /var/log/messages:

Sep 24 18:06:59 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: Account HEAD returning 503 for [] (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:06:59 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012    fp, sock = self._parent_class_getter()#012  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txe63709eb31c94cd88949b-005422ec22) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: Account HEAD returning 503 for [] (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:00 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txb72b0ffad16b4b69bf1dc-005422ec24) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: Account HEAD returning 503 for [] (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:02 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx176d8372cd584f6aafc28-005422ec26) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: Account HEAD returning 503 for [] (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:06 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txc6efdf36b254465cb6e5d-005422ec2a) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: Account HEAD returning 503 for [] (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)
Sep 24 18:07:14 zeus proxy-server: STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 239, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012    fp, sock = self.create()#012  File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 192, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: txddd4071c6912449188af1-005422ec32) (client_ip: 192.168.10.2)

I guess there is maybe a problem with memcached? (as suggested by ebyenjoy)

MEMCACHE CONFIG:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 192.168.10.2" # Here I set management network IP, not storage network... Is it valid?

OK, that log has been solved..

memcached service has to be launched before swift services. Ok, now... When trying "swift stat":

Sep 24 20:33:03 zeus proxy-server: Auth Token confirmed use of v2.0 apis
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.4:6002/sdb3 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: ERROR with Account server 192.168.20.3:6002/sdb2 re: Trying to HEAD /v1/AUTH_37d8b1fb62374ef9a5642f992f624eec: Connection refused (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)
Sep 24 20:33:03 zeus proxy-server: Account HEAD returning 503 for [] (txn: txd4d81079a93741c2aedc2-0054230e5e) (client_ip: 192.168.10.2)

At the end... Connection refused.

If I netstat by 11211 port on Controller node (where memcached is suposed to be listening) i get:

[root@zeus admin]# netstat -putona | grep 11211
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2100/memcached      off (0.00/0/0)
tcp        0      0 192.168.10.2:46290          192.168.10.2:11211          ESTABLISHED 2739/python         off (0.00/0/0)
tcp        0      0 192.168.10.2:11211          192.168.10.2:46290          ESTABLISHED 2100/memcached      keepalive (6964,61/0/0)
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2100/memcached      off (0.00/0/0)

So I guess is listening... Then... why controller refuses connection?

EDITED:

Ok, fixed errors on /var/log/messages by adding and changing bind_ip with storage network IP in proxy-server.conf (on controller) and container, account and object conf files on Storage nodes.

But, still same issue when swift stat (connecion refused).

[root@zeus admin]# swift stat 
[Errno 111] Connection refused

Now, I don't know how to track errors. Any idea?

EDITED ON REPLY TO guimaluf:

If the connections is been refused, probably means your rsyncd is not properly configured.

Is rsyncd running? Paste rsyncd.conf and check if ports and IP are correct set.

RSYNCD.conf on Storage Node 1:

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.20.3 # storage network IP
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

RSYNCD.conf on Storage Node 2:

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.20.4
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

Which ports are open? Run netstat -lvpn and check it out.

NETSTAT ON PROXY NODE

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 192.168.10.2:11211          0.0.0.0:*                   LISTEN      2102/memcached      
tcp        0      0 0.0.0.0:9292                0.0.0.0:*                   LISTEN      2535/python         
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1799/rpcbind        
tcp        0      0 192.168.20.2:8080           0.0.0.0:*                   LISTEN      4936/python         
tcp        0      0 127.0.0.1:5939              0.0.0.0:*                   LISTEN      1963/teamviewerd    
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2119/sshd           
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1964/cupsd          
tcp        0      0 0.0.0.0:48056               0.0.0.0:*                   LISTEN      1936/rpc.statd      
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2384/master         
tcp        0      0 0.0.0.0:35357               0.0.0.0:*                   LISTEN      5021/python         
tcp        0      0 0.0.0.0:6080                0.0.0.0:*                   LISTEN      2639/python         
tcp        0      0 0.0.0.0:9696                0.0.0.0:*                   LISTEN      2508/python         
tcp        0      0 0.0.0.0:8773                0.0.0.0:*                   LISTEN      2607/python         
tcp        0      0 0.0.0.0:8774                0.0.0.0:*                   LISTEN      2607/python         
tcp        0      0 0.0.0.0:8775                0.0.0.0:*                   LISTEN      2607/python         
tcp        0      0 0.0.0.0:9191                0.0.0.0:*                   LISTEN      2576/python         
tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN      5021/python         
tcp        0      0 0.0.0.0:8776                0.0.0.0:*                   LISTEN      2516/python         
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2437/qpidd          
tcp        0      0 192.168.10.2:3306           0.0.0.0:*                   LISTEN      2288/mysqld         
tcp        0      0 :::111                      :::*                        LISTEN      1799/rpcbind        
tcp        0      0 :::80                       :::*                        LISTEN      2416/httpd          
tcp        0      0 :::22                       :::*                        LISTEN      2119/sshd           
tcp        0      0 ::1:631                     :::*                        LISTEN      1964/cupsd          
tcp        0      0 ::1:25                      :::*                        LISTEN      2384/master         
tcp        0      0 :::33026                    :::*                        LISTEN      1936/rpc.statd      
tcp        0      0 :::5672                     :::*                        LISTEN      2437/qpidd          
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1799/rpcbind        
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1964/cupsd          
udp        0      0 192.168.20.2:123            0.0.0.0:*                               2127/ntpd           
udp        0      0 192.168.10.2:123            0.0.0.0:*                               2127/ntpd           
udp        0      0 10.51.1.116:123             0.0.0.0:*                               2127/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2127/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2127/ntpd           
udp        0      0 0.0.0.0:702                 0.0.0.0:*                               1799/rpcbind        
udp        0      0 0.0.0.0:53440               0.0.0.0:*                               1936/rpc.statd      
udp        0      0 0.0.0.0:840                 0.0.0.0:*                               1936/rpc.statd      
udp        0      0 192.168.10.2:11211          0.0.0.0:*                               2102/memcached      
udp        0      0 :::111                      :::*                                    1799/rpcbind        
udp        0      0 fe80::20e:cff:fe6b:d84a:123 :::*                                    2127/ntpd           
udp        0      0 fe80::219:d1ff:fe8f:fc54:123 :::*                                    2127/ntpd           
udp        0      0 ::1:123                     :::*                                    2127/ntpd           
udp        0      0 :::123                      :::*                                    2127/ntpd           
udp        0      0 :::702                      :::*                                    1799/rpcbind        
udp        0      0 :::43488                    :::*                                    1936/rpc.statd      
netstat: no support for `AF INET (sctp)' on this system.
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     13422  1999/hald           @/var/run/hald/dbus-iZZEOzrwgR
unix  2      [ ACC ]     STREAM     LISTENING     23747  3248/gnome-panel    /tmp/orbit-admin/linc-cb0-0-58b876da5c3ef
unix  2      [ ACC ]     STREAM     LISTENING     24117  3288/bonobo-activat /tmp/orbit-admin/linc-cd8-0-58a5d3b9a6ee9
unix  2      [ ACC ]     STREAM     LISTENING     24291  3314/gconf-helper   /tmp/orbit-admin/linc-cf2-0-5bd3f8c8b738d
unix  2      [ ACC ]     STREAM     LISTENING     24344  3310/trashapplet    /tmp/orbit-admin/linc-cee-0-3becc4c5e9047
unix  2      [ ACC ]     STREAM     LISTENING     24348  3311/wnck-applet    /tmp/orbit-admin/linc-cef-0-17c90e97e91c2
unix  2      [ ACC ]     STREAM     LISTENING     13378  1990/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     25067  3375/nm-applet      /tmp/orbit-admin/linc-d2f-0-1fe57b670c33
unix  2      [ ACC ]     STREAM     LISTENING     25115  3437/gnome-screensa /tmp/orbit-admin/linc-d49-0-28275bb8b8d95
unix  2      [ ACC ]     STREAM     LISTENING     25326  3417/gpk-update-ico /tmp/orbit-admin/linc-d59-0-7e09410ed4fca
unix  2      [ ACC ]     STREAM     LISTENING     25353  3346/evolution-alar /tmp/orbit-admin/linc-d12-0-7ed02077657c
unix  2      [ ACC ]     STREAM     LISTENING     25478  3357/gnome-power-ma /tmp/orbit-admin/linc-d1d-0-6ed384796b800
unix  2      [ ACC ]     STREAM     LISTENING     25514  3279/nautilus       /tmp/orbit-admin/linc-ccf-0-1dd0bb857d0e5
unix  2      [ ACC ]     STREAM     LISTENING     25748  3445/gdm-user-switc /tmp/orbit-admin/linc-d75-0-2e32c8844a7bd
unix  2      [ ACC ]     STREAM     LISTENING     25769  3448/notification-a /tmp/orbit-admin/linc-d78-0-611c45154d3e5
unix  2      [ ACC ]     STREAM     LISTENING     25789  3449/clock-applet   /tmp/orbit-admin/linc-d79-0-59f173a853607
unix  2      [ ACC ]     STREAM     LISTENING     25999  3616/gconf-im-setti /tmp/orbit-admin/linc-e20-0-7b66f0f8de66d
unix  2      [ ACC ]     STREAM     LISTENING     26039  3440/gnote          /tmp/orbit-admin/linc-d70-0-48b152a6cb167
unix  2      [ ACC ]     STREAM     LISTENING     26147  3639/wineserver     socket
unix  2      [ ACC ]     STREAM     LISTENING     24238  3247/pulseaudio     /tmp/.esd-500/socket
unix  2      [ ACC ]     STREAM     LISTENING     24242  3247/pulseaudio     /home/admin/.pulse/705c70fcec307615a8c014ef00000038-runtime/native
unix  2      [ ACC ]     STREAM     LISTENING     8867   1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     558953 10451/gedit         /tmp/gedit.admin.623590353
unix  2      [ ACC ]     STREAM     LISTENING     14962  2288/mysqld         /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     17234  2741/Xorg           @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     15186  2384/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     15193  2384/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     15197  2384/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     15201  2384/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     15205  2384/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     15357  2408/abrtd          /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     348033 5126/firefox        /tmp/orbit-admin/linc-1406-0-78ca38b34445
unix  2      [ ACC ]     STREAM     LISTENING     22839  3137/gnome-session  @/tmp/.ICE-unix/3137
unix  2      [ ACC ]     STREAM     LISTENING     558961 10451/gedit         /tmp/orbit-admin/linc-28d3-0-12ac38b78759d
unix  2      [ ACC ]     STREAM     LISTENING     15209  2384/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     15213  2384/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     17235  2741/Xorg           /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     22840  3137/gnome-session  /tmp/.ICE-unix/3137
unix  2      [ ACC ]     STREAM     LISTENING     17399  2738/gdm-simple-sla @/tmp/gdm-greeter-INYMRsSV
unix  2      [ ACC ]     STREAM     LISTENING     22865  3220/gconfd-2       /tmp/orbit-admin/linc-c94-0-5ea0540a3bdd8
unix  2      [ ACC ]     STREAM     LISTENING     17584  2738/gdm-simple-sla @/tmp/gdm-session-TlcsJVrd
unix  2      [ ACC ]     STREAM     LISTENING     13296  1964/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     23119  3137/gnome-session  /tmp/orbit-admin/linc-c41-0-6760a38d472e6
unix  2      [ ACC ]     STREAM     LISTENING     15423  2416/httpd          /etc/httpd/run/wsgi.2416.0.1.sock
unix  2      [ ACC ]     STREAM     LISTENING     15217  2384/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     15221  2384/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     15225  2384/master         private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     15229  2384/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     23328  3127/gnome-keyring- /tmp/orbit-admin/linc-c37-0-2384310c9a33e
unix  2      [ ACC ]     STREAM     LISTENING     12888  1799/rpcbind        /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     23334  3127/gnome-keyring- /tmp/keyring-fZ8CMy/socket.ssh
unix  2      [ ACC ]     STREAM     LISTENING     23343  3127/gnome-keyring- /tmp/keyring-fZ8CMy/socket.pkcs11
unix  2      [ ACC ]     STREAM     LISTENING     23344  3228/gnome-settings /tmp/orbit-admin/linc-c9c-0-624ae00a9b096
unix  2      [ ACC ]     STREAM     LISTENING     15233  2384/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     15237  2384/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     13070  1917/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     24750  3353/bluetooth-appl /tmp/orbit-admin/linc-d19-0-61cf19df820bd
unix  2      [ ACC ]     STREAM     LISTENING     15241  2384/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     22681  3146/dbus-daemon    @/tmp/dbus-LVuDSvuzZX
unix  2      [ ACC ]     STREAM     LISTENING     15245  2384/master         private/retry
unix  2      [ ACC ]     STREAM     LISTENING     31434  3724/gnome-terminal /tmp/orbit-admin/linc-e8c-0-6d27754be7f85
unix  2      [ ACC ]     STREAM     LISTENING     13415  1999/hald           @/var/run/hald/dbus-IYPDQvbH8x
unix  2      [ ACC ]     STREAM     LISTENING     22363  3127/gnome-keyring- /tmp/keyring-fZ8CMy/socket
unix  2      [ ACC ]     STREAM     LISTENING     23560  3243/metacity       /tmp/orbit-admin/linc-cab-0-786355d72e1f9
unix  2      [ ACC ]     STREAM     LISTENING     23611  3230/seahorse-daemo /tmp/orbit-admin/linc-c9e-0-60a259b333ae6
unix  2      [ ACC ]     STREAM     LISTENING     15249  2384/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     15253  2384/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     15257  2384/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     15261  2384/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     15265  2384/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     15269  2384/master         private/scache
netstat: no support for `AF IPX' on this system.
netstat: no support for `AF AX25' on this system.
netstat: no support for `AF X25' on this system.
netstat: no support for `AF NETROM' on this system.

NETSTAT ON STORAGE NODE 1

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      2036/rpcbind        
tcp        0      0 192.168.20.3:6000           0.0.0.0:*                   LISTEN      14360/python        
tcp        0      0 192.168.20.3:6001           0.0.0.0:*                   LISTEN      14354/python        
tcp        0      0 192.168.20.3:6002           0.0.0.0:*                   LISTEN      14355/python        
tcp        0      0 127.0.0.1:5939              0.0.0.0:*                   LISTEN      1902/teamviewerd    
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2345/sshd           
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      2194/cupsd          
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2441/master         
tcp        0      0 0.0.0.0:3260                0.0.0.0:*                   LISTEN      2320/tgtd           
tcp        0      0 0.0.0.0:59517               0.0.0.0:*                   LISTEN      2167/rpc.statd      
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2476/qpidd          
tcp        0      0 :::111                      :::*                        LISTEN      2036/rpcbind        
tcp        0      0 :::22                       :::*                        LISTEN      2345/sshd           
tcp        0      0 ::1:631                     :::*                        LISTEN      2194/cupsd          
tcp        0      0 ::1:25                      :::*                        LISTEN      2441/master         
tcp        0      0 :::3260                     :::*                        LISTEN      2320/tgtd           
tcp        0      0 :::5672                     :::*                        LISTEN      2476/qpidd          
tcp        0      0 :::873                      :::*                        LISTEN      2353/xinetd         
tcp        0      0 :::37385                    :::*                        LISTEN      2167/rpc.statd      
udp        0      0 0.0.0.0:58221               0.0.0.0:*                               2167/rpc.statd      
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               2036/rpcbind        
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               2194/cupsd          
udp        0      0 10.51.1.117:123             0.0.0.0:*                               2361/ntpd           
udp        0      0 192.168.20.3:123            0.0.0.0:*                               2361/ntpd           
udp        0      0 192.168.11.3:123            0.0.0.0:*                               2361/ntpd           
udp        0      0 192.168.10.3:123            0.0.0.0:*                               2361/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2361/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2361/ntpd           
udp        0      0 0.0.0.0:647                 0.0.0.0:*                               2167/rpc.statd      
udp        0      0 0.0.0.0:939                 0.0.0.0:*                               2036/rpcbind        
udp        0      0 :::111                      :::*                                    2036/rpcbind        
udp        0      0 fe80::f82c:21ff:fe51:2f6:123 :::*                                    2361/ntpd           
udp        0      0 fe80::21c:c0ff:fe06:7428:123 :::*                                    2361/ntpd           
udp        0      0 fe80::38f1:a6ff:fec8:22d:123 :::*                                    2361/ntpd           
udp        0      0 fe80::4010:82ff:fe2d:df8:123 :::*                                    2361/ntpd           
udp        0      0 fe80::248:54ff:fe4c:2440:123 :::*                                    2361/ntpd           
udp        0      0 ::1:123                     :::*                                    2361/ntpd           
udp        0      0 :::123                      :::*                                    2361/ntpd           
udp        0      0 :::939                      :::*                                    2036/rpcbind        
udp        0      0 :::48743                    :::*                                    2167/rpc.statd      
netstat: no support for `AF INET (sctp)' on this system.
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     22321  1459/ovs-vswitchd   /var/run/openvswitch/br-tun.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     19170  2751/Xorg           /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     22214  2937/gnome-session  /tmp/.ICE-unix/2937
unix  2      [ ACC ]     STREAM     LISTENING     17328  2441/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     23041  3088/gconfd-2       /tmp/orbit-gdm/linc-c10-0-5d28750446c53
unix  2      [ ACC ]     STREAM     LISTENING     17332  2441/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     23115  2937/gnome-session  /tmp/orbit-gdm/linc-b79-0-5ed1d1874eb3c
unix  2      [ ACC ]     STREAM     LISTENING     24246  3153/gnome-settings /tmp/orbit-gdm/linc-c51-0-54a27fe94f33d
unix  2      [ ACC ]     STREAM     LISTENING     24579  3161/bonobo-activat /tmp/orbit-gdm/linc-c59-0-6c508cf82554c
unix  2      [ ACC ]     STREAM     LISTENING     24585  3151/at-spi-registr /tmp/orbit-gdm/linc-c4f-0-7587a01325aec
unix  2      [ ACC ]     STREAM     LISTENING     26013  3287/metacity       /tmp/orbit-gdm/linc-cd7-0-528450cb67547
unix  2      [ ACC ]     STREAM     LISTENING     22323  1459/ovs-vswitchd   /var/run/openvswitch/br-tun.snoop
unix  2      [ ACC ]     STREAM     LISTENING     26839  3372/gnome-power-ma /tmp/orbit-gdm/linc-d2c-0-378b94a92404e
unix  2      [ ACC ]     STREAM     LISTENING     19269  2544/python         /var/lib/neutron/metadata_proxy
unix  2      [ ACC ]     STREAM     LISTENING     13993  1459/ovs-vswitchd   /var/run/openvswitch/br-ex.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     27081  3392/polkit-gnome-a /tmp/orbit-gdm/linc-d40-0-1911b5bfc2
unix  2      [ ACC ]     STREAM     LISTENING     27120  3418/gdm-simple-gre /tmp/orbit-gdm/linc-d5a-0-586cc91363e23
unix  2      [ ACC ]     STREAM     LISTENING     13995  1459/ovs-vswitchd   /var/run/openvswitch/br-ex.snoop
unix  2      [ ACC ]     STREAM     LISTENING     16215  2228/hald           @/var/run/hald/dbus-BiG0S6mixe
unix  2      [ ACC ]     STREAM     LISTENING     30075  3536/pulseaudio     /var/lib/gdm/.pulse/95aea038884b2057483add8800000039-runtime/native
unix  2      [ ACC ]     STREAM     LISTENING     16164  2219/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     8634   1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     16820  2320/tgtd           /var/run/tgtd.ipc_abstract_namespace.0
unix  2      [ ACC ]     STREAM     LISTENING     19169  2751/Xorg           @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     17336  2441/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     17340  2441/master         private/scache
unix  2      [ ACC ]     STREAM     LISTENING     17257  2441/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     17264  2441/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     20516  2720/gdm-simple-sla @/tmp/gdm-greeter-AeCcMcTu
unix  2      [ ACC ]     STREAM     LISTENING     17268  2441/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     21226  2931/dbus-daemon    @/tmp/dbus-CmjxSqH8lP
unix  2      [ ACC ]     STREAM     LISTENING     17272  2441/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     17276  2441/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     17280  2441/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     17284  2441/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     16086  2194/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     16208  2228/hald           @/var/run/hald/dbus-N9C6uB5ZGa
unix  2      [ ACC ]     STREAM     LISTENING     17288  2441/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     17292  2441/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     15552  2036/rpcbind        /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     17296  2441/master         private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     17300  2441/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     17429  2465/abrtd          /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     21762  2720/gdm-simple-sla @/tmp/gdm-session-NjSEEfOG
unix  2      [ ACC ]     STREAM     LISTENING     13609  1448/ovsdb-server   /var/run/openvswitch/db.sock
unix  2      [ ACC ]     STREAM     LISTENING     13611  1448/ovsdb-server   /var/run/openvswitch/ovsdb-server.1448.ctl
unix  2      [ ACC ]     STREAM     LISTENING     17304  2441/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     17308  2441/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     17312  2441/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     17316  2441/master         private/retry
unix  2      [ ACC ]     STREAM     LISTENING     22213  2937/gnome-session  @/tmp/.ICE-unix/2937
unix  2      [ ACC ]     STREAM     LISTENING     17320  2441/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     13651  1459/ovs-vswitchd   /var/run/openvswitch/ovs-vswitchd.1459.ctl
unix  2      [ ACC ]     STREAM     LISTENING     17324  2441/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     15749  2148/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     13689  1459/ovs-vswitchd   /var/run/openvswitch/br-int.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     13691  1459/ovs-vswitchd   /var/run/openvswitch/br-int.snoop

NETSTAT ON STORAGE NODE 2

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:59694               0.0.0.0:*                   LISTEN      2246/rpc.statd      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      2116/rpcbind        
tcp        0      0 192.168.20.4:6000           0.0.0.0:*                   LISTEN      15841/python        
tcp        0      0 192.168.20.4:6001           0.0.0.0:*                   LISTEN      15835/python        
tcp        0      0 192.168.20.4:6002           0.0.0.0:*                   LISTEN      15836/python        
tcp        0      0 127.0.0.1:5939              0.0.0.0:*                   LISTEN      1978/teamviewerd    
tcp        0      0 192.168.122.1:53            0.0.0.0:*                   LISTEN      2796/dnsmasq        
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2401/sshd           
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      2274/cupsd          
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2497/master         
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2547/qpidd          
tcp        0      0 :::36075                    :::*                        LISTEN      2246/rpc.statd      
tcp        0      0 :::111                      :::*                        LISTEN      2116/rpcbind        
tcp        0      0 :::22                       :::*                        LISTEN      2401/sshd           
tcp        0      0 ::1:631                     :::*                        LISTEN      2274/cupsd          
tcp        0      0 ::1:25                      :::*                        LISTEN      2497/master         
tcp        0      0 :::5672                     :::*                        LISTEN      2547/qpidd          
tcp        0      0 :::873                      :::*                        LISTEN      2409/xinetd         
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               2116/rpcbind        
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               2274/cupsd          
udp        0      0 192.168.122.1:123           0.0.0.0:*                               2417/ntpd           
udp        0      0 192.168.20.4:123            0.0.0.0:*                               2417/ntpd           
udp        0      0 192.168.11.4:123            0.0.0.0:*                               2417/ntpd           
udp        0      0 192.168.10.4:123            0.0.0.0:*                               2417/ntpd           
udp        0      0 10.51.1.118:123             0.0.0.0:*                               2417/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2417/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2417/ntpd           
udp        0      0 0.0.0.0:1019                0.0.0.0:*                               2116/rpcbind        
udp        0      0 192.168.122.1:53            0.0.0.0:*                               2796/dnsmasq        
udp        0      0 0.0.0.0:54072               0.0.0.0:*                               2246/rpc.statd      
udp        0      0 0.0.0.0:67                  0.0.0.0:*                               2796/dnsmasq        
udp        0      0 0.0.0.0:726                 0.0.0.0:*                               2246/rpc.statd      
udp        0      0 :::111                      :::*                                    2116/rpcbind        
udp        0      0 fe80::8c6d:e3ff:feb5:b97:123 :::*                                    2417/ntpd           
udp        0      0 fe80::3815:16ff:fe7b:47b:123 :::*                                    2417/ntpd           
udp        0      0 fe80::3c08:8aff:fe0b:435:123 :::*                                    2417/ntpd           
udp        0      0 fe80::a838:9cff:fe21:998:123 :::*                                    2417/ntpd           
udp        0      0 fe80::cc48:cfff:feca:d62:123 :::*                                    2417/ntpd           
udp        0      0 fe80::e07d:ffff:fe83:ad9:123 :::*                                    2417/ntpd           
udp        0      0 fe80::a0ad:d0ff:fe71:700:123 :::*                                    2417/ntpd           
udp        0      0 fe80::88ca:b1ff:fef2:966:123 :::*                                    2417/ntpd           
udp        0      0 ::1:123                     :::*                                    2417/ntpd           
udp        0      0 fe80::7271:bcff:fe94:97f:123 :::*                                    2417/ntpd           
udp        0      0 fe80::201:2ff:feb0:ebd3:123 :::*                                    2417/ntpd           
udp        0      0 :::123                      :::*                                    2417/ntpd           
udp        0      0 :::1019                     :::*                                    2116/rpcbind        
udp        0      0 :::60244                    :::*                                    2246/rpc.statd      
netstat: no support for `AF INET (sctp)' on this system.
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     20025  3103/dbus-daemon    @/tmp/dbus-0QyG3GyUun
unix  2      [ ACC ]     STREAM     LISTENING     16959  2497/master         private/scache
unix  2      [ ACC ]     STREAM     LISTENING     16026  2299/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     19231  2910/Xorg           /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     16070  2308/hald           @/var/run/hald/dbus-mm8mSFIP04
unix  2      [ ACC ]     STREAM     LISTENING     20102  3105/gnome-session  /tmp/.ICE-unix/3105
unix  2      [ ACC ]     STREAM     LISTENING     20169  3112/gconfd-2       /tmp/orbit-gdm/linc-c28-0-1e5e07d9934f7
unix  2      [ ACC ]     STREAM     LISTENING     14388  1629/ovs-vswitchd   /var/run/openvswitch/ovs-vswitchd.1629.ctl
unix  2      [ ACC ]     STREAM     LISTENING     20244  3105/gnome-session  /tmp/orbit-gdm/linc-c21-0-6bc2794398d3f
unix  2      [ ACC ]     STREAM     LISTENING     20342  3131/gnome-settings /tmp/orbit-gdm/linc-c3b-0-753fe0e8dbfbc
unix  2      [ ACC ]     STREAM     LISTENING     20426  3133/bonobo-activat /tmp/orbit-gdm/linc-c3d-0-4d6994157b8a2
unix  2      [ ACC ]     STREAM     LISTENING     20432  3126/at-spi-registr /tmp/orbit-gdm/linc-c36-0-539548d07ba6d
unix  2      [ ACC ]     STREAM     LISTENING     20510  3141/metacity       /tmp/orbit-gdm/linc-c45-0-19bf99e0354cc
unix  2      [ ACC ]     STREAM     LISTENING     21578  3143/gdm-simple-gre /tmp/orbit-gdm/linc-c47-0-72c89a3d6ca8c
unix  2      [ ACC ]     STREAM     LISTENING     21679  3146/polkit-gnome-a /tmp/orbit-gdm/linc-c4a-0-5997457a8f913
unix  2      [ ACC ]     STREAM     LISTENING     21748  3144/gnome-power-ma /tmp/orbit-gdm/linc-c48-0-1ce089c0b26d8
unix  2      [ ACC ]     STREAM     LISTENING     8818   1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     19890  2904/gdm-simple-sla @/tmp/gdm-greeter-pxNMOhtW
unix  2      [ ACC ]     STREAM     LISTENING     20067  2904/gdm-simple-sla @/tmp/gdm-session-lduoMDHu
unix  2      [ ACC ]     STREAM     LISTENING     19230  2910/Xorg           @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     17663  2607/libvirtd       /var/run/libvirt/libvirt-sock
unix  2      [ ACC ]     STREAM     LISTENING     17665  2607/libvirtd       /var/run/libvirt/libvirt-sock-ro
unix  2      [ ACC ]     STREAM     LISTENING     17052  2521/abrtd          /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     16876  2497/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     14419  1629/ovs-vswitchd   /var/run/openvswitch/br-int.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     16883  2497/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     16887  2497/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     16891  2497/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     14421  1629/ovs-vswitchd   /var/run/openvswitch/br-int.snoop
unix  2      [ ACC ]     STREAM     LISTENING     14346  1618/ovsdb-server   /var/run/openvswitch/db.sock
unix  2      [ ACC ]     STREAM     LISTENING     14348  1618/ovsdb-server   /var/run/openvswitch/ovsdb-server.1618.ctl
unix  2      [ ACC ]     STREAM     LISTENING     22733  1629/ovs-vswitchd   /var/run/openvswitch/br-tun.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     22735  1629/ovs-vswitchd   /var/run/openvswitch/br-tun.snoop
unix  2      [ ACC ]     STREAM     LISTENING     20101  3105/gnome-session  @/tmp/.ICE-unix/3105
unix  2      [ ACC ]     STREAM     LISTENING     16895  2497/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     16899  2497/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     16903  2497/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     16907  2497/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     15948  2274/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     16911  2497/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     16915  2497/master         private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     22163  3159/pulseaudio     /var/lib/gdm/.pulse/25fbee1e357459864436a51500000037-runtime/native
unix  2      [ ACC ]     STREAM     LISTENING     15520  2116/rpcbind        /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     16919  2497/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     16923  2497/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     16927  2497/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     16931  2497/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     16063  2308/hald           @/var/run/hald/dbus-JjnLasFYOG
unix  2      [ ACC ]     STREAM     LISTENING     15724  2227/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     16935  2497/master         private/retry
unix  2      [ ACC ]     STREAM     LISTENING     16939  2497/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     16943  2497/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     16947  2497/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     16951  2497/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     16955  2497/master         private/anvil

There is any firewall running? Is it blocking ports 873, 6000, 6001, 6002?

Nope, Iptables is disabled for testing first... :S

========

Ive just realize your problem it's happening also on proxy-node.

What endpoints are set on keystone? Also run swift --debug stat and paste for us

Ok, endpoints:

[root@zeus admin]# keystone endpoint-list
+----------------------------------+-----------+----------------------------------------+----------------------------------------+-----------------------------------+----------------------------------+
|                id                |   region  |               publicurl                |              internalurl               |              adminurl             |            service_id            |
+----------------------------------+-----------+----------------------------------------+----------------------------------------+-----------------------------------+----------------------------------+
| 4a0847cf7f9c4dc8a73bf76b63fa6a4f | regionOne |   http://zeus:8774/v2/%(tenant_id)s    |   http://zeus:8774/v2/%(tenant_id)s    | http://zeus:8774/v2/%(tenant_id)s | 7863d4a9dce648129315239e41c6a1ea |
| 8d84a88672f946828e4bd971b4215d61 | regionOne |         http://zeus:5000/v2.0          |         http://zeus:5000/v2.0          |       http://zeus:35357/v2.0      | 959fa3a77d8745a98c93f3e3a232b182 |
| 980ec695bc6942c8b866d7f04d125d26 | regionOne |            http://zeus:9696            |            http://zeus:9696            |          http://zeus:9696         | a30a90c5d4c140038d67c5144cf01214 |
| a9b5a188c79546e89beecb6a13841dd3 | regionOne | http://zeus:8080/v1/AUTH_%(tenant_id)s | http://zeus:8080/v1/AUTH_%(tenant_id)s |          http://zeus:8080         | a175bed646d647d2820728230155d9ce |
| bed17d64ed8c42b783ca7cf457cc63ed | regionOne |   http://zeus:8776/v1/%(tenant_id)s    |   http://zeus:8776/v1/%(tenant_id)s    | http://zeus:8776/v1/%(tenant_id)s | 60d60cc313a24f829d2cc24590864d78 |
| d97540ab45aa4ba299d4a7b228ca97e4 | regionOne |            http://zeus:9292            |            http://zeus:9292            |          http://zeus:9292         | fd09daf22ef145199795d2e118aca09c |
| e2b179f7370d45e0bf73aa7e39c9b8d5 | regionOne |   http://zeus:8776/v2/%(tenant_id)s    |   http://zeus:8776/v2/%(tenant_id)s    | http://zeus:8776/v2/%(tenant_id)s | 0d7a707e91ff44439bdaedae32f7b1d2 |
+----------------------------------+-----------+----------------------------------------+----------------------------------------+-----------------------------------+----------------------------------+

Now, swift --debug stat log:

DEBUG:keystoneclient.session:REQ: curl -i -X POST http://zeus:35357/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "passadmin"}}}'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 6819
DEBUG:keystoneclient.session:RESP: [200] {'date': 'Thu, 25 Sep 2014 12:36:27 GMT', 'content-type': 'application/json', 'content-length': '6819', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-09-25T12:36:27.032625", "expires": "2014-09-25T13:36:27Z", "id": "MIIMKwYJKoZIhvcNAQcCoIIMHDCCDBgCAQExCTAHBgUrDgMCGjCCCoEGCSqGSIb3DQEHAaCCCnIEggpueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wOS0yNVQxMjozNjoyNy4wMzI2MjUiLCAiZXhwaXJlcyI6ICIyMDE0LTA5LTI1VDEzOjM2OjI3WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIkFkbWluIFRlbmFudCIsICJlbmFibGVkIjogdHJ1ZSwgImlkIjogIjM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgIm5hbWUiOiAiYWRtaW4ifX0sICJzZXJ2aWNlQ2F0YWxvZyI6IFt7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly96ZXVzOjg3NzQvdjIvMzdkOGIxZmI2MjM3NGVmOWE1NjQyZjk5MmY2MjRlZWMiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vemV1czo4Nzc0L3YyLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgImlkIjogIjM1NDNhNTBmNWNkZDQyNTVhZjg3NjUxODhmNjdlMTI4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vemV1czo4Nzc0L3YyLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovL3pldXM6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly96ZXVzOjk2OTYiLCAiaWQiOiAiYTg2OWJmNzE2ZDdiNDU0OTllNjNhMTg1OTQ3ZDZhMjMiLCAicHVibGljVVJMIjogImh0dHA6Ly96ZXVzOjk2OTYifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAibmV0d29yayIsICJuYW1lIjogIm5ldXRyb24ifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czo4Nzc2L3YyLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92Mi8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyIsICJpZCI6ICI1Yzg3NjM5YmQ5YmU0MTNjODA3M2ZkOTYyZGY5NjIwMiIsICJwdWJsaWNVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92Mi8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWV2MiIsICJuYW1lIjogImNpbmRlcnYyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovL3pldXM6OTI5MiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly96ZXVzOjkyOTIiLCAiaWQiOiAiMDgwYmVjMmUwZmRmNGZmMzgyMGQ4Zjc5MjM5MWQ0NTUiLCAicHVibGljVVJMIjogImh0dHA6Ly96ZXVzOjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czo4Nzc2L3YxLzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92MS8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyIsICJpZCI6ICI4YTJlYjg2NGIwYmY0ODkyYTBhYTIwOTQxZjhjMWJiZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovL3pldXM6ODc3Ni92MS8zN2Q4YjFmYjYyMzc0ZWY5YTU2NDJmOTkyZjYyNGVlYyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAibmFtZSI6ICJjaW5kZXIifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czo4MDgwIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6ODA4MC92MS9BVVRIXzM3ZDhiMWZiNjIzNzRlZjlhNTY0MmY5OTJmNjI0ZWVjIiwgImlkIjogIjMzNDk5YzczNTU5ZTRlZDlhMTZjYjNhY2Q3MmRjOGYzIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vemV1czo4MDgwL3YxL0FVVEhfMzdkOGIxZmI2MjM3NGVmOWE1NjQyZjk5MmY2MjRlZWMifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAib2JqZWN0LXN0b3JlIiwgIm5hbWUiOiAic3dpZnQifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vemV1czozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovL3pldXM6NTAwMC92Mi4wIiwgImlkIjogIjAwZTc3NTAxNWY3OTRiZDliZGUzNThiMTc2MDFhMmQyIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vemV1czo1MDAwL3YyLjAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaWRlbnRpdHkiLCAibmFtZSI6ICJrZXlzdG9uZSJ9XSwgInVzZXIiOiB7InVzZXJuYW1lIjogImFkbWluIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICI2NzJmOWZjMzAzZDQ0Yjk1OTE5NjczYjE0OTIzZWUwMCIsICJyb2xlcyI6IFt7Im5hbWUiOiAiX21lbWJlcl8ifSwgeyJuYW1lIjogImFkbWluIn1dLCAibmFtZSI6ICJhZG1pbiJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5ZmUyZmY5ZWU0Mzg0YjE4OTRhOTA4NzhkM2U5MmJhYiIsICJiYWQ4NDc3MGNiMWY0NjIzYmMxNGVjOTBjYTk5N2QyYiJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHuVuancCsf3LP+RQMrGQ8y6VSrUXq+tfj-vFhKeWs2Admm0aAnWt4yeusw6V4svw2QLkxh6ZG582yevtW02T4tDF86zVIPrcn2xYuhNhp0m6FC3n-Cbhd-LVmUDfT4RaFqitU1mjQ8Lgqycc+umgKJPOW9--k63gxdz1TlFufoBTM5xAPwGXK47WjQ9cRpw-CAYXpOwQEVV5iOZDOJzmx6J5B2+dxoJTImQXdoUMeeCadvcuwCo8bO5iDPtxjerA3wJhjA6JFIwHhpUX04Z0j7MNAQlk16dgJc-mXlHmr5GsmjkVEuUKyxAuDrA2wwRoaNjO4D9O-8P+10FC9fJ0oE=", "tenant": {"description": "Admin Tenant", "enabled": true, "id": "37d8b1fb62374ef9a5642f992f624eec", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://zeus:8774/v2/37d8b1fb62374ef9a5642f992f624eec", "region": "regionOne", "internalURL": "http://zeus:8774/v2/37d8b1fb62374ef9a5642f992f624eec", "id": "3543a50f5cdd4255af8765188f67e128", "publicURL": "http://zeus:8774/v2/37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://zeus:9696", "region": "regionOne", "internalURL": "http://zeus:9696", "id": "a869bf716d7b45499e63a185947d6a23", "publicURL": "http://zeus:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://zeus:8776/v2/37d8b1fb62374ef9a5642f992f624eec", "region": "regionOne", "internalURL": "http://zeus:8776/v2/37d8b1fb62374ef9a5642f992f624eec", "id": "5c87639bd9be413c8073fd962df96202", "publicURL": "http://zeus:8776/v2/37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://zeus:9292", "region": "regionOne", "internalURL": "http://zeus:9292", "id": "080bec2e0fdf4ff3820d8f792391d455", "publicURL": "http://zeus:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://zeus:8776/v1/37d8b1fb62374ef9a5642f992f624eec", "region": "regionOne", "internalURL": "http://zeus:8776/v1/37d8b1fb62374ef9a5642f992f624eec", "id": "8a2eb864b0bf4892a0aa20941f8c1bbe", "publicURL": "http://zeus:8776/v1/37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://zeus:8080", "region": "regionOne", "internalURL": "http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec", "id": "33499c73559e4ed9a16cb3acd72dc8f3", "publicURL": "http://zeus:8080/v1/AUTH_37d8b1fb62374ef9a5642f992f624eec"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://zeus:35357/v2.0", "region": "regionOne", "internalURL": "http://zeus:5000/v2.0", "id": "00e775015f794bd9bde358b17601a2d2", "publicURL": "http://zeus:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "672f9fc303d44b95919673b14923ee00", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "bad84770cb1f4623bc14ec90ca997d2b"]}}}

DEBUG:iso8601.iso8601:Parsed 2014-09-25T13:36:27Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'13', 'daydash': u'25', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'27', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'09', 'day': None, 'minute': u'36'} with default timezone <iso8601.iso8601.Utc object at 0x222a9d0>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'09' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 9 for 'month' with default 9
DEBUG:iso8601.iso8601:Got u'25' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 25 for 'day' with default 25
DEBUG:iso8601.iso8601:Got u'13' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'36' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'27' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
INFO:urllib3.connectionpool:Starting new HTTP connection (1): zeus
ERROR:swiftclient:[Errno 111] Connection refused
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1208, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 485, in head_account
    conn.request(method, parsed.path, '', headers)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 213, in request
    files=files, **self.requests_args)
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 181, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 206, in send
    raise ConnectionError(sockerr)
ConnectionError: [Errno 111] Connection refused
[Errno 111] Connection refused