Ask Your Question
1

swift stat Account HEAD failed 503 Internal Server Error

asked 2014-11-19 07:16:35 -0500

Yogesh Devi gravatar image

updated 2014-11-19 09:14:32 -0500

Hello - I have installed a new multi node swift - as follows

  • keystone on = swift4 = 160.110.73.161
  • proxy server on = swift2 = 160.110.73.176
  • Object storage nodes on swift1= 160.110.73.224 and swift3 = 160.110.73.200

When I do a

$swift stat

I get following error

**Account HEAD failed: http://swift2:8080:8080/v1/AUTH_a7256141431e4f52ad80fb828f7b116c 503 Internal Server Error**

This error seems to be rather common place and I fixed most of the configuration based on inputs from similar problems posted in this forum. However I continue to get this error. Below are the concerned lines from the log of my proxy server ( /var/log/messages) corresponding to one "swift stat" request . I have set log level to "DEBUG"

Nov 19 13:33:08 swift2 proxy-server: Authenticating user token
Nov 19 13:33:08 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:08 swift2 proxy-server: Storing token in cache
Nov 19 13:33:08 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:08 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf39b69b9d01c43a69b583-00546ce264)
Nov 19 13:33:08 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf39b69b9d01c43a69b583-00546ce264) (client_ip: 160.110.73.176)
Nov 19 13:33:09 swift2 proxy-server: Authenticating user token
Nov 19 13:33:09 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:09 swift2 proxy-server: Returning cached token
Nov 19 13:33:09 swift2 proxy-server: Storing token in cache
Nov 19 13:33:09 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:09 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: txf18b3ae34b05432fa0002-00546ce265)
Nov 19 13:33:09 swift2 proxy-server: Account HEAD returning 503 for [] (txn: txf18b3ae34b05432fa0002-00546ce265) (client_ip: 160.110.73.176)
Nov 19 13:33:11 swift2 proxy-server: Authenticating user token
Nov 19 13:33:11 swift2 proxy-server: Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
Nov 19 13:33:11 swift2 proxy-server: Returning cached token
Nov 19 13:33:11 swift2 proxy-server: Storing token in cache
Nov 19 13:33:11 swift2 <135>proxy-server: Received request from user: 09f43bb78fa44151b9f6a773076ef045 with project_id : a7256141431e4f52ad80fb828f7b116c and roles: admin,_member_
Nov 19 13:33:11 swift2 proxy-server: ERROR with Account server 160.110.73.176:6003/sdb re: Trying to HEAD /v1/AUTH_a7256141431e4f52ad80fb828f7b116c: Connection refused (txn: tx7cc06bd2dc4741549d45d-00546ce267)
Nov 19 13:33:11 swift2 proxy-server: Account HEAD returning 503 for [] (txn: tx7cc06bd2dc4741549d45d-00546ce267) (client_ip: 160.110.73.176)
Nov 19 ...
(more)
edit retag flag offensive close merge delete

Comments

Apologies for long post - for the impatient expert - please see the end of the post on my account ring and the port number error - and enlighten me on possible configuration goof up ...

Yogesh Devi gravatar imageYogesh Devi ( 2014-11-19 09:43:31 -0500 )edit

I guess the issue is there with your object storage ERROR with Account server 160.110.73.176:6003/sdb. The error points out so.

ebyenjoys gravatar imageebyenjoys ( 2014-11-19 11:09:55 -0500 )edit

This is unrelated to the problem, but Keystone users often end with that double-port thing (:8080:8080). That is invalid and needs fixing, but I don't know just how you accomplished that, so I have no suggestions.

zaitcev gravatar imagezaitcev ( 2015-01-16 10:23:19 -0500 )edit

3 answers

Sort by » oldest newest most voted
1

answered 2014-11-19 15:24:07 -0500

echiu gravatar image

updated 2015-01-14 14:38:47 -0500

smaffulli gravatar image

See this section on 503 errors in Troubleshooting Swifthttps://wiki.openstack.org/wiki/Troub...

503 or 507 Errors

If you see specific problems like 503 Service Unavailable, double-check the port values to ensure they match the value in etc/swift/auth-server.conf. When adding a new account/user, the auth server calls out to the proxy server based on the default_cluster_url config file value and it should match the bind_port value. You also have to configure the port for the proxy-server.conf under the [filter:auth] section, which defaults to 11000.

edit flag offensive delete link more
0

answered 2015-01-16 10:21:48 -0500

zaitcev gravatar image

You can see that proxy tries to connect to swift2 (and gets connection refused), while the question's premise states that storage nodes are swift1 and swift3. Sounds like ring does not match where the servers are started. You can examine the actual ring with "swift-ring-builder account.builder". Without additional arguments it would print the ring. Make sure it contains the 2 storage nodes and not the proxy node, make sure that storage nodes do actually run the account/container/object servers, and that ports listed in the ring match the configurations of the running servers.

edit flag offensive delete link more
0

answered 2015-12-12 13:54:41 -0500

https://bugs.launchpad.net/openstack-manuals/+bug/1525563 (https://bugs.launchpad.net/openstack-...)

/etc/swift/account-server.conf
/etc/swift/container-server.conf
/etc/swift/object-server.conf

Modify files above on storage nodes to have

[pipeline:main]
...
pipeline = recon account-server

and append

[filter:recon]
use = egg:swift#recon
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-11-19 07:16:35 -0500

Seen: 5,934 times

Last updated: Jan 16 '15