Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

swift stat Account HEAD failed 503 Internal Server Error

Hello - I have installed a new multi node swift - as follows

  • keystone on = swift4 = 160.110.73.161
  • proxy server on = swift2 = 160.110.73.176
  • Object storage nodes on swift1= 160.110.73.224 and swift3 = 160.110.73.200

When I do a

$swift stat

I get following error

**Account HEAD failed: http://swift2:8080:8080/v1/AUTH_a7256141431e4f52ad80fb828f7b116c 503 Internal Server Error**

This error seems to be rather common place and I fixed most of the configuration based on inputs from similar problems posted in this forum. However I continue to get this error. Below are the concerned lines from the log of my proxy server ( /var/log/messages) corresponding to one "swift stat" request . I have set log level to "DEBUG"

Nov 19 13:33:08 swift2 proxy-server: Authenticating user token
Nov 19 13:33:08 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:08 swift2 proxy-server: Storing token in cache
Nov 19 13:33:08 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:08 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf39b69b9d01c43a69b583-00546ce264)
Nov 19 13:33:08 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf39b69b9d01c43a69b583-00546ce264) (client_ip: 160.110.73.176)
Nov 19 13:33:09 swift2 proxy-server: Authenticating user token
Nov 19 13:33:09 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:09 swift2 proxy-server: Returning cached token
Nov 19 13:33:09 swift2 proxy-server: Storing token in cache
Nov 19 13:33:09 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:09 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf18b3ae34b05432fa0002-00546ce265)
Nov 19 13:33:09 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf18b3ae34b05432fa0002-00546ce265) (client_ip: 160.110.73.176)
Nov 19 13:33:11 swift2 proxy-server: Authenticating user token
Nov 19 13:33:11 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:11 swift2 proxy-server: Returning cached token
Nov 19 13:33:11 swift2 proxy-server: Storing token in cache
Nov 19 13:33:11 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:11 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx7cc06bd2dc4741549d45d-00546ce267)
Nov 19 13:33:11 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx7cc06bd2dc4741549d45d-00546ce267) (client_ip: 160.110.73.176)
Nov 19 13:33:15 swift2 proxy-server: Authenticating user token
Nov 19 13:33:15 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:15 swift2 proxy-server: Returning cached token
Nov 19 13:33:15 swift2 proxy-server: Storing token in cache
Nov 19 13:33:15 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:15 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txb9b2e589e4cf43368d02a-00546ce26b)
Nov 19 13:33:16 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txb9b2e589e4cf43368d02a-00546ce26b) (client_ip: 160.110.73.176)
Nov 19 13:33:23 swift2 proxy-server: Authenticating user token
Nov 19 13:33:23 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:23 swift2 proxy-server: Returning cached token
Nov 19 13:33:23 swift2 proxy-server: Storing token in cache
Nov 19 13:33:23 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:23 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx1ad4e721d0724f54bd212-00546ce273)
Nov 19 13:33:24 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx1ad4e721d0724f54bd212-00546ce273) (client_ip: 160.110.73.176)

Below are my configuration files

proxy-server.conf - on swift2 ( 160.110.73.176)

[DEFAULT]
bind_port = 8080
user = swift
bind_ip = 0.0.0.0
expose_info = true

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
set log_name = proxy-server
set log_level = DEBUG
set log_name = proxy-server
set log_facility = LOG_LOCAL0

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = swift4
delay_auth_decision = true
auth_port = 35357
auth_protocol = http
auth_uri = http://160.110.73.161:5000
admin_tenant_name = admin
admin_user = admin
admin_password = admin123
signing_dir = /tmp/keystone-signing-swift

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 160.110.73.176:11211

The account-server.conf on (on swift1= 160.110.73.224 and swift3 = 160.110.73.200)

[DEFAULT]
user = swift
bind_ip = 0.0.0.0

[pipeline:main]
pipeline = healthcheck recon account-server

[app:account-server]
use = egg:swift#account
set log_name = account-server
set log_facility = LOG_LOCAL0
set log_level = DEBUG

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:recon]
use = egg:swift#recon

swift stat Account HEAD failed 503 Internal Server Error

Hello - I have installed a new multi node swift - as follows

  • keystone on = swift4 = 160.110.73.161
  • proxy server on = swift2 = 160.110.73.176
  • Object storage nodes on swift1= 160.110.73.224 and swift3 = 160.110.73.200

When I do a

$swift stat

I get following error

**Account HEAD failed: http://swift2:8080:8080/v1/AUTH_a7256141431e4f52ad80fb828f7b116c 503 Internal Server Error**

This error seems to be rather common place and I fixed most of the configuration based on inputs from similar problems posted in this forum. However I continue to get this error. Below are the concerned lines from the log of my proxy server ( /var/log/messages) corresponding to one "swift stat" request . I have set log level to "DEBUG"

Nov 19 13:33:08 swift2 proxy-server: Authenticating user token
Nov 19 13:33:08 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:08 swift2 proxy-server: Storing token in cache
Nov 19 13:33:08 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:08 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf39b69b9d01c43a69b583-00546ce264)
Nov 19 13:33:08 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf39b69b9d01c43a69b583-00546ce264) (client_ip: 160.110.73.176)
Nov 19 13:33:09 swift2 proxy-server: Authenticating user token
Nov 19 13:33:09 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:09 swift2 proxy-server: Returning cached token
Nov 19 13:33:09 swift2 proxy-server: Storing token in cache
Nov 19 13:33:09 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:09 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf18b3ae34b05432fa0002-00546ce265)
Nov 19 13:33:09 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf18b3ae34b05432fa0002-00546ce265) (client_ip: 160.110.73.176)
Nov 19 13:33:11 swift2 proxy-server: Authenticating user token
Nov 19 13:33:11 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:11 swift2 proxy-server: Returning cached token
Nov 19 13:33:11 swift2 proxy-server: Storing token in cache
Nov 19 13:33:11 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:11 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx7cc06bd2dc4741549d45d-00546ce267)
Nov 19 13:33:11 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx7cc06bd2dc4741549d45d-00546ce267) (client_ip: 160.110.73.176)
Nov 19 13:33:15 swift2 proxy-server: Authenticating user token
Nov 19 13:33:15 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:15 swift2 proxy-server: Returning cached token
Nov 19 13:33:15 swift2 proxy-server: Storing token in cache
Nov 19 13:33:15 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:15 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txb9b2e589e4cf43368d02a-00546ce26b)
Nov 19 13:33:16 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txb9b2e589e4cf43368d02a-00546ce26b) (client_ip: 160.110.73.176)
Nov 19 13:33:23 swift2 proxy-server: Authenticating user token
Nov 19 13:33:23 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:23 swift2 proxy-server: Returning cached token
Nov 19 13:33:23 swift2 proxy-server: Storing token in cache
Nov 19 13:33:23 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:23 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx1ad4e721d0724f54bd212-00546ce273)
Nov 19 13:33:24 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx1ad4e721d0724f54bd212-00546ce273) (client_ip: 160.110.73.176)

Below are my configuration files

proxy-server.conf - on swift2 ( 160.110.73.176)

[DEFAULT]
bind_port = 8080
user = swift
bind_ip = 0.0.0.0
expose_info = true

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
set log_name = proxy-server
set log_level = DEBUG
set log_name = proxy-server
set log_facility = LOG_LOCAL0

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = swift4
delay_auth_decision = true
auth_port = 35357
auth_protocol = http
auth_uri = http://160.110.73.161:5000
admin_tenant_name = admin
admin_user = admin
admin_password = admin123
signing_dir = /tmp/keystone-signing-swift

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 160.110.73.176:11211

The account-server.conf on (on swift1= 160.110.73.224 and swift3 = 160.110.73.200)

[DEFAULT]
user = swift
bind_ip = 0.0.0.0

[pipeline:main]
pipeline = healthcheck recon account-server

[app:account-server]
use = egg:swift#account
set log_name = account-server
set log_facility = LOG_LOCAL0
set log_level = DEBUG

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:recon]
use = egg:swift#recon

I also noticed another strange thing - looks like the port number 6003 is indeed in there

When I check my account builder --

   $ swift-ring-builder account.builder

I see

account.builder, build version 1
262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  region  zone      ip address  port  replication ip  replication port      name weight partitions balance meta
             0       1     1  160.110.73.176  6003  160.110.73.176              6030       sdb   1.00     786432    0.00

I wonder how did I end up with port 6003 ?

swift stat Account HEAD failed 503 Internal Server Error

Hello - I have installed a new multi node swift - as follows

  • keystone on = swift4 = 160.110.73.161
  • proxy server on = swift2 = 160.110.73.176
  • Object storage nodes on swift1= 160.110.73.224 and swift3 = 160.110.73.200

When I do a

$swift stat

I get following error

**Account HEAD failed: http://swift2:8080:8080/v1/AUTH_a7256141431e4f52ad80fb828f7b116c 503 Internal Server Error**

This error seems to be rather common place and I fixed most of the configuration based on inputs from similar problems posted in this forum. However I continue to get this error. Below are the concerned lines from the log of my proxy server ( /var/log/messages) corresponding to one "swift stat" request . I have set log level to "DEBUG"

Nov 19 13:33:08 swift2 proxy-server: Authenticating user token
Nov 19 13:33:08 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:08 swift2 proxy-server: Storing token in cache
Nov 19 13:33:08 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:08 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf39b69b9d01c43a69b583-00546ce264)
Nov 19 13:33:08 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf39b69b9d01c43a69b583-00546ce264) (client_ip: 160.110.73.176)
Nov 19 13:33:09 swift2 proxy-server: Authenticating user token
Nov 19 13:33:09 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:09 swift2 proxy-server: Returning cached token
Nov 19 13:33:09 swift2 proxy-server: Storing token in cache
Nov 19 13:33:09 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:09 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf18b3ae34b05432fa0002-00546ce265)
Nov 19 13:33:09 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf18b3ae34b05432fa0002-00546ce265) (client_ip: 160.110.73.176)
Nov 19 13:33:11 swift2 proxy-server: Authenticating user token
Nov 19 13:33:11 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:11 swift2 proxy-server: Returning cached token
Nov 19 13:33:11 swift2 proxy-server: Storing token in cache
Nov 19 13:33:11 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:11 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx7cc06bd2dc4741549d45d-00546ce267)
Nov 19 13:33:11 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx7cc06bd2dc4741549d45d-00546ce267) (client_ip: 160.110.73.176)
Nov 19 13:33:15 swift2 proxy-server: Authenticating user token
Nov 19 13:33:15 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:15 swift2 proxy-server: Returning cached token
Nov 19 13:33:15 swift2 proxy-server: Storing token in cache
Nov 19 13:33:15 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:15 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txb9b2e589e4cf43368d02a-00546ce26b)
Nov 19 13:33:16 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txb9b2e589e4cf43368d02a-00546ce26b) (client_ip: 160.110.73.176)
Nov 19 13:33:23 swift2 proxy-server: Authenticating user token
Nov 19 13:33:23 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:23 swift2 proxy-server: Returning cached token
Nov 19 13:33:23 swift2 proxy-server: Storing token in cache
Nov 19 13:33:23 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:23 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx1ad4e721d0724f54bd212-00546ce273)
Nov 19 13:33:24 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx1ad4e721d0724f54bd212-00546ce273) (client_ip: 160.110.73.176)

Below are my configuration files

proxy-server.conf - on swift2 ( 160.110.73.176)

[DEFAULT]
bind_port = 8080
user = swift
bind_ip = 0.0.0.0
expose_info = true

[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
set log_name = proxy-server
set log_level = DEBUG
set log_name = proxy-server
set log_facility = LOG_LOCAL0

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = swift4
delay_auth_decision = true
auth_port = 35357
auth_protocol = http
auth_uri = http://160.110.73.161:5000
admin_tenant_name = admin
admin_user = admin
admin_password = admin123
signing_dir = /tmp/keystone-signing-swift

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 160.110.73.176:11211

The account-server.conf on (on swift1= 160.110.73.224 and swift3 = 160.110.73.200)

[DEFAULT]
user = swift
bind_ip = 0.0.0.0

[pipeline:main]
pipeline = healthcheck recon account-server

[app:account-server]
use = egg:swift#account
set log_name = account-server
set log_facility = LOG_LOCAL0
set log_level = DEBUG

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:recon]
use = egg:swift#recon

I also noticed another strange thing - looks like the port number 6003 is indeed in there in my configuration ( NOTE: below might be just a red herring - so use your judgement when reading :-)

When I check my account builder --

   $ swift-ring-builder account.builder

I see ouput as follows

account.builder, build version 1
262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  region  zone      ip address  port  replication ip  replication port      name weight partitions balance meta
             0       1     1  160.110.73.176  6003  160.110.73.176              6030       sdb   1.00     786432    0.00

I wonder how suppose when I build the account ring and added a zone to it - I specify port number for my zone --- as far as I remember I did I end it like this

   $ swift-ring-builder /etc/swift/account.builder add z1-160.110.73.176:6003R160.110.73.176:6030/sdb 1

Or is it likely that I would have done a typo and ended up with port # 6003 ?... ? Now the question that I have are as follows ( pardon me they are daft - I confess that I am a SWIFT newbee )

  1. Question 1 - The IP address that I specify to the above swift-ring-builder command , is the ip of my proxy server - right ...
  2. Question 2 in that case on my proxy server - After I start only the proxy server process ( /etc/init.d/openstack-swift-proxy start) I should see someone listening on this port right - Or I also need to start some other swift processes on proxy server ? - I observed that after I start my proxy server process I see no process is listening on either 6003 or 6030 ports
  3. Question 3 - Since I have nothing against port 6003 and will happily live with it - what configuration change should I do and where to ensure that this port is used by the account server and I don't get this error