Ask Your Question
0

503 Service Unavailable

asked 2010-09-29 18:23:04 -0500

ryan-rodd gravatar image

Hey guys, I've done my best to figure this one out, including re-installing the OS and Swift multiple times, but I keep running into this error after trying to create a test user.

swift-auth-add-user -K (mypwd) -a test tester testing swift-auth-add-user -K (mypwd) -a test tester testing -c /etc/swift/auth-server.conf

both result in: Update failed: 503 Service Unavailable

UPDATE: Additionally, the syslog returns the following error for this event: auth-server ERROR Unhandled exception in ReST request: Connection refused

I've verified that all swift services are running and I've altered the configuration files many times to no avail. Any suggestions?

edit retag flag offensive close merge delete

7 answers

Sort by ยป oldest newest most voted
0

answered 2010-10-01 20:19:27 -0500

ryan-rodd gravatar image

Thanks for that. You're right in that I was using steps from SAIO for virtual devices. Turning mount_check=false has seemed to have fixed things.

The thing is, I actually am mounting a separate device (hard drive) whose path I am specifying (/dev/sdb1 is mounted at /mnt/sdb1). However when I had mount_check=true and devices=/dev/sdb1 the services were telling me the device was not mounted. Why was that?

And FYI I'm adding devices to the rings like this: ... swift-ring-builder object.builder add z1-127.0.0.1:6000/sdb1 1 ...

Thanks again

edit flag offensive delete link more
0

answered 2010-10-01 20:45:09 -0500

gholt gravatar image

Ah, devices= is supposed to be set to the directory where a set of mount points can be found. The system is set up to allow many drives per server. For instance, we often have systems with devices=/srv/node and if you did an ls /srv/node you might see:

sda sdc sde sdg sdi sdk sdm sdo sdq sds sdu sdw sdb sdd sdf sdh sdj sdl sdn sdp sdr sdt sdv sdx

Each one of those being a separate mount point and therefore device.

I'm pretty sure you don't want devices=/dev/sdb1 but you probably don't want devices=/mnt either. By convention, you might want to create an /srv/node directory and mount sdb1 at /srv/node/sdb1 and set devices=/srv/node

With devices=/dev/sdb1 I'm not sure what might be happening on your system. It might actually be trying to write files to /dev/sdb1/sdb1/* If you're running as root, it might actually be succeeding! Best to double check things aren't wonky in that respect; you may need to do a bit of cleanup.

edit flag offensive delete link more
0

answered 2010-10-01 19:28:50 -0500

gholt gravatar image

Ah, the 507 returned by the account server in there indicates the account server thinks the device is not mounted.

If you're using "pretend" devices as with the standard SAIO, you'll want to ensure mount_check = false is in your account-server.conf file (or your account-server/*.conf files) under the [DEFAULT] section(s).

If that's not set right in your account server configurations, you'll want to double check your container and object server configurations too.

edit flag offensive delete link more
0

answered 2010-10-01 17:15:14 -0500

ryan-rodd gravatar image

Thanks for setting me straight on this. This has definitely helped me go groom all my config files. I'm going to post one more problem at the expense of looking like a total noob, but the 503 error has re-spawn in a different form. The syslog now returns:

auth-server validate_token('AUTH_tk93fb6c4577bd4432ad43666a75044b29', _, _) = (86399.916866064072, '.super_admin', '.single_use', '.reseller_admin') [0.05] auth-server 127.0.0.1 - - [01/Oct/2010:17:04:10 +0000] "GET /token/AUTH_tk93fb6c4577bd4432ad43666a75044b29 HTTP/1.0" 204 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.0488 account-server 127.0.0.1 - - [01/Oct/2010:17:04:10 +0000] "PUT /sdb1/80201/AUTH_f4748dba45464d2f9730e1c81174d887" 507 - "tx71ec71de-ff88-4bc1-a9df-593a72d257fc" "-" "-" 0.0012 "" proxy-server Account PUT returning 503 for [503], transaction tx71ec71de-ff88-4bc1-a9df-593a72d257fc proxy-server - 127.0.0.1 01/Oct/2010/17/04/10 PUT /v1/AUTH_f4748dba45464d2f9730e1c81174d887 HTTP/1.0 503 - - .super_admin%3A.single_use%2CAUTH_tk93fb6c4577bd4432ad43666a75044b29 - - - tx71ec71de-ff88-4bc1-a9df-593a72d257fc - 0.0085 auth-server ERROR attempting to create account http://127.0.0.1:80/v1/AUTH_f4748dba45464d2f9730e1c81174d887 (http://127.0.0.1:80/v1/AUTH_f4748dba4...) : 503 Internal Server Error auth-server FAILED create_user('test', 'tester', _, True, False) [0.15] auth-server 127.0.0.1 - - [01/Oct/2010:17:04:10 +0000] "PUT /account/test/tester HTTP/1.0" 503 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.1477

It looks like the error is sent by the account-server to the proxy server in response to the PUT. Shed any light on this one? Thanks again. You guys rock.

edit flag offensive delete link more
0

answered 2010-09-29 22:19:05 -0500

gholt gravatar image

Ah, okay. That error is now the proxy server trying to connect back to the auth server for token validation. If you're auth server is configured to run on port 80, you'll need to set port=80 in your proxy-server.conf under the [filter:auth] section. It defaults to 11000, which I think is what the auth server defaults to.

edit flag offensive delete link more
0

answered 2010-09-29 20:48:48 -0500

ryan-rodd gravatar image

Thanks for the response! I've gone back and checked the bound ports for each service. They are:

account-server.conf: 6003 container-server.conf: 6002 object-server.conf: 6001 and for the important ones: auth-server.conf: 80 - default_cluster_url = http://127.0.0.1:8080/v1 proxy-server.conf: 8080

This has gotten me past the 503 error (thanks) but has created another one. Now when issuing the same command (swift-auth-add-user -K cumulo -a test tester testing) the terminal hangs and I'm forced after a minute or so to interrupt (command-c) it. The terminal spits out a traceback:

Traceback (most recent call last): File "/usr/bin/swift-auth-add-user", line 69, in <module> resp = conn.getresponse() File "/usr/lib/pymodules/python2.6/swift/common/bufferedhttp.py", line 101, in getresponse response = HTTPConnection.getresponse(self) File "/usr/lib/python2.6/httplib.py", line 986, in getresponse response.begin() File "/usr/lib/python2.6/httplib.py", line 391, in begin version, status, reason = self._read_status() File "/usr/lib/python2.6/httplib.py", line 349, in _read_status line = self.fp.readline() File "/usr/lib/python2.6/socket.py", line 406, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/pymodules/python2.6/eventlet/greenio.py", line 228, in recv timeout_exc=socket.timeout("timed out")) ...

And this is recorded in the system log:

STDOUT: Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/eventlet/wsgi.py", line 335, in handle_one_response result = self.application(self.environ, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/healthcheck.py", line 38, in __call__ return self.app(env, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/memcache.py", line 31, in __call__ return self.app(env, start_response) File "/usr/lib/pymodules/python2.6/swift/common/middleware/auth.py", line 61, in __call__ self.get_groups(token, memcache_client=cache_from_env(env)) File "/usr/lib/pymodules/python2.6/swift/common/middleware/auth.py", line 131, in get_groups '/token/%s' % token, ssl=self.ssl) File "/usr/lib/pymodules/python2.6/swift/common/bufferedhttp.py", line 169, in http_connect_raw conn.endheaders() File "/usr/lib/python2.6/httplib.py", line 904, in endheaders self._send_output()#012 File "/usr/lib/python2.6/httplib.py", line 776, in _send_output self.send(msg)#012 File "/usr/lib/python2.6/httplib.py", line 735, in send self.connect() File "/usr/lib/pymodules/python2.6/swift/common/bufferedhttp.py", line 80, in connect return HTTPConnection.connect(self)#012 File "/usr/lib/python2.6/httplib.py", line 716, in connect self.timeout) File "/usr/lib/pymodules/python2.6/eventlet/green/socket.py", line 92, in create_connection raise error, msg error: [Errno 111] ECONNREFUSED

I will admit this could have been caused by my endless tinkering to settings and the such so I'm tempted to just go back and reinstall swift and ubuntu (without xubuntu-desktop). Thanks again.

edit flag offensive delete link more
0

answered 2010-09-29 19:35:37 -0500

gholt gravatar image

When adding a new account/user, the auth server will call out to the proxy server based on the default_cluster_url config file value (defaults to http://127.0.0.1:8080/v1 I think). Might double-check that value in /etc/swift/auth-server.conf and that you're running the proxy server on that same port (bind_port value, I think that defaults to port 80).

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2010-09-29 18:23:04 -0500

Seen: 204 times

Last updated: Oct 01 '10