Ask Your Question

503 error running multiple-server

asked 2011-06-17 11:11:49 -0500

shengjie-min gravatar image

I I've been following the instruction on ( . I have three storage and a proxy server running as 4 vms(host only). proxy server: storage 1 : storage 2 : storage 3 :

ring builds perfectly, rebalanced perfectly. rm -f .builder *.ring.gz backups/.builder backups/*.ring.gz

swift-ring-builder object.builder create 18 3 1 swift-ring-builder object.builder add z1- 1 swift-ring-builder object.builder add z2- 1 swift-ring-builder object.builder add z3- 1 swift-ring-builder object.builder rebalance

swift-ring-builder container.builder create 18 3 1 swift-ring-builder container.builder add z1- 1 swift-ring-builder container.builder add z2- 1 swift-ring-builder container.builder add z3- 1 swift-ring-builder container.builder rebalance

swift-ring-builder account.builder create 18 3 1 swift-ring-builder account.builder add z1- 1 swift-ring-builder account.builder add z2- 1 swift-ring-builder account.builder add z3- 1 swift-ring-builder account.builder rebalance

When I was trying to start proxy server, tempauth was not working, and then I noticed swauth is still installed with in swift(I can see all the swauth commands files, like swauth-prep, swauth-add-user under /usr/bin), I've changed my proxy-server.conf as below

[DEFAULT] #cert_file = /etc/swift/cert.crt #key_file = /etc/swift/cert.key bind_port = 8080 workers = 8 user = swift

[pipeline:main] pipeline = healthcheck cache swauth proxy-server

[app:proxy-server] use = egg:swift#proxy allow_account_management = true

[filter:swauth] use = egg:swift#swauth default_swift_cluster = local# super_admin_key = swauthkey

[filter:healthcheck] use = egg:swift#healthcheck

[filter:cache] use = egg:swift#memcache

memcache_servers =

then when i try to run

root@ubuntu:/etc/swift# swauth-prep -K swauthkey

Auth subsystem prep failed: 500 Server Error

I get this error in sys.log

Jun 17 04:08:32 ubuntu proxy-server Account PUT returning 503 for [503, 503, 503] (txn: tx9fadfec6-1f85-4e8f-9982-2987bc44465d) Jun 17 04:08:32 ubuntu proxy-server - - 17/Jun/2011/11/08/32 PUT /v1/AUTH_.auth HTTP/1.0 503 - - - - - - tx9fadfec6-1f85-4e8f-9982-2987bc44465d - 0.0125 Jun 17 04:08:32 ubuntu proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last):#012 File "/usr/lib/pymodules/python2.6/swift/common/middleware/", line 320, in handle#012 return self.handle_request(req)(env, start_response)#012 File "/usr/lib/pymodules/python2.6/swift/common/middleware/", line 383, in handle_request#012 req.response = handler(req)#012 File "/usr/lib/pymodules/python2.6/swift/common/middleware/", line 402, in handle_prep#012 (path, resp.status))#012Exception: Could not create the main auth account: /v1/AUTH_.auth 503 Internal Server Error#012: {'SCRIPT_NAME': '/auth/v2', 'webob.adhoc_attrs': {'start_time': 1308308912.1579649, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'POST', 'PATH_INFO': '/.prep ... (more)

edit retag flag offensive close merge delete

12 answers

Sort by ยป oldest newest most voted

answered 2013-07-14 01:54:27 -0500

juhu2 gravatar image

I also meet this problem when running "swift stat". I find some error in the /var/log/message:

swift Node error limited (node) (txn: tx395f4fc90fb44e368aace4db659045d4) swift Account HEAD returning 503 for [] (txn: tx395f4fc90fb44e368aace4db659045d4) (client_ip:

By reading the reply of gholt (gholt) said on 2011-06-17, I try to append "mount_check = false" to the config file of account-server.conf, container-server.conf, object-server.conf. Then this issue is disappear.

Thand for gholt.

edit flag offensive delete link more

answered 2011-06-17 16:16:14 -0500

gholt gravatar image

There is work planned to improve logging and documentation, etc. But all that takes time. We mostly throw stack traces because, though they're ugly, they tend to help more than "error reading file". :)

The HTTP codes are almost always standard, or at least /real/ close. 507 means "Insufficient Storage" and though we're using for "Unmounted" that's a pretty close match.

The only non-standard codes I'm aware Swift logs are:

498 Rate-Limiting In Effect 499 Client Disconnected Early

edit flag offensive delete link more

answered 2011-06-17 14:54:32 -0500

shengjie-min gravatar image

Thanks very much, guys.

xfs_repaired all sda3, restarted the storages, reloaded the proxy server, wola, working!

so the error messages made me worried. If you guys didn't tell me 507 means something wrong with the device, I wouldn't even have had the chance to figure out this message:

Jun 17 06:36:18 ubuntu account-server - - [17/Jun/2011:13:36:18 +0000] "HEAD /sda3/138541/AUTH_.auth" 507 - "tx6f3c1dbb-cb91-44cb-ab40-02bebc461fe7" "-" "-" 0.0010 ""

Generally, I found the error msg and logging are not up 2 a very good standard. Is that because the nature of Python, or it's the way the exceptions/errors got handled? we have plans to enhance them? Also the response code, like 201, 204, 500, 503, 507 etc etc, they are not standard http code, right? Are they defined internally in SWIFT somewhere? or we can have a error code menu to check when we running into issues again next time?

edit flag offensive delete link more

answered 2011-06-17 14:22:28 -0500

gholt gravatar image

Yeah, definitely something up with either hardware or the kernel version or something. You'll want to re-enable that mount_check btw; it is a nice safety feature so you don't end up writing data to your os drive. :)

edit flag offensive delete link more

answered 2011-06-17 14:19:10 -0500

shengjie-min gravatar image

I think I am close to the issue now

root@ubuntu:/srv/node# ls -lsa ls: cannot access sda3 total 8 4 drwxr-xr-x 3 swift swift 4096 2011-06-14 01:45 . 4 drwxr-xr-x 3 root root 4096 2011-06-14 01:45 .. ? d????????? ? ? ? ? ? sda3

root@ubuntu:/srv/node/sda3# ls -lsa ls: cannot open directory .: Input/output error

root@ubuntu:/etc/swift# xfs_check /srv/node/sda3 xfs_check: cannot open /srv/node/sda3: Input/output error

@gholt: account-server.conf [DEFAULT] bind_ip = workers = 2 mount_check=false

[pipeline:main] pipeline = account-server

[app:account-server] use = egg:swift#account




edit flag offensive delete link more

answered 2011-06-17 14:08:35 -0500

shengjie-min gravatar image

@gholt, I've tried put mount_check = false in all the conf files you mentioned. Now getting this in sys.log on the account servers.

Jun 17 07:05:16 ubuntu account-server ERROR __call__ error with PUT /sda3/138541/AUTH_.auth : [Errno 5] Input/output error: '/srv/node/sda3/accounts/138541' (txn: tx43000d3d-75de-4bf7-8d4e-9aa6cff4d5df) Jun 17 07:05:16 ubuntu account-server - - [17/Jun/2011:14:05:16 +0000] "PUT /sda3/138541/AUTH_.auth" 500 798 "tx43000d3d-75de-4bf7-8d4e-9aa6cff4d5df" "-" "-" 0.0023 "" Jun 17 07:05:21 ubuntu kernel: [190617.489812] Filesystem "sda3": xfs_log_force: error 5 returned.

Proxy server error looks the same.

edit flag offensive delete link more

answered 2011-06-17 14:06:49 -0500

gholt gravatar image

Can you paste your account-server.conf and perhaps also an "ls -la /srv/node/sda3" on one of the account servers?

edit flag offensive delete link more

answered 2011-06-17 13:56:43 -0500

shengjie-min gravatar image

actually, they are mounted, three of them are all mounted

root@ubuntu:/etc/swift# mount /srv/node/sda3 mount: /dev/sda3 already mounted or /srv/node/sda3 busy mount: according to mtab, /dev/sda3 is already mounted on /srv/node/sda3

From Gparted, All three storage have the same:

Partition: /dev/sda3
File System: xfs
Mount Point:/srv/node/sda3
Size: 4.99GiB Used: 17.69MiB Unused: 4.98GiB

edit flag offensive delete link more

answered 2011-06-17 13:55:14 -0500

gholt gravatar image

If sda3 isn't really a disk on its own but is instead just a directory on another disk (such as if you were setting up a test system and not an actual production system) you can disable the mount checking with:

mount_check = false

Each in each of these configs:

account-server.conf container-server.conf object-server.conf

edit flag offensive delete link more

answered 2011-06-17 13:43:37 -0500

notmyname gravatar image

the 507 on the account server indicates that the drive sda3 is unmounted

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2011-06-17 11:11:49 -0500

Seen: 113 times

Last updated: Jul 14 '13