Ask Your Question
0

Get object failed with 503 error

asked 2013-06-27 03:59:37 -0500

249524324-q gravatar image

Hi,guys! I've got a 503 error code in swift log while getting objects by Multiple client.I trace the log trying to figure out the problem of the issue.but there are some information make me confused.I pickued up relational part in proxy log.

> Jun 26 11:57:34 station215 proxy-server ERROR with Object server 192.168.52.180:6020/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:35 station215 proxy-server ERROR with Object server 192.168.52.181:6010/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:36 station215 proxy-server ERROR with Object server 192.168.52.186:6030/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:37 station215 proxy-server ERROR with Object server 192.168.52.182:6040/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:38 station215 proxy-server ERROR with Object server 192.168.52.47:6010/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:39 station215 proxy-server ERROR with Object server 192.168.52.186:6040/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:39 station215 proxy-server Object GET returning 503 for [] (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187) Jun 26 11:57:39 station215 proxy-server 192.168.52.187 192.168.52.187 26/Jun/2013/03/57/39 GET /v1/AUTH_mwliaotest/linux/67344 HTTP/1.0 503 - Jakarta%20Commons-HttpClient/3.1 mwliaotest%2CAUTH_tkda034cde76e746da8ad31303856d2cbd - 118 - tx9c2b55167f244dd785487-0051ca66ad - 6.3996 -

As far as I know,Proxy-Server will look up the location of object server in the ring.gz file.But,the proxy-server route the GET request to all of 6 object-servers.I config the replacas_count with 3.Why did the proxy-server route request to the other 3 object-server?

edit retag flag offensive close merge delete

4 answers

Sort by ยป oldest newest most voted
0

answered 2013-06-27 11:39:34 -0500

249524324-q gravatar image

I do not really understand.Is it useful to connect the other object server ?There ara no replica stored in those object.Why did the proxy do this?I appreciate.

edit flag offensive delete link more
0

answered 2013-06-27 06:31:56 -0500

Because your didn't get a response from obj server with 200 or 404.

Proxy should get enough 200 or enough 404 from obj server and convert it into 200 or 404 to user (you). All of your nodes are timeout, so proxy request all nodes then.

edit flag offensive delete link more
0

answered 2013-06-28 00:28:00 -0500

249524324-q gravatar image

Thanks Samuel Merritt, that solved my question.

edit flag offensive delete link more
0

answered 2013-06-27 16:03:48 -0500

torgomatic gravatar image

Well, here's an example for you. Let's say your primary nodes are d0, d1, and d2, with handoff nodes d3, d4, and d5.

On object PUT, the proxy will normally write 3 replicas to d0, d1, and d2, as you said. However, if d0 is full, then the object server for d0 will return a 507 response, and the proxy will use a handoff, so the object gets written to nodes d1, d2, d3.

Similarly, if a machine is down, then more handoffs will be used. Let's say d0 and d1 are full, and d2's machine has a broken network port. Now the object PUT will get written to d3, d4, and d5: all handoffs.

The proxy looks to some handoffs in order to give read-after-write consistency for new objects. Otherwise, if an object were written to only handoff nodes, then a subsequent GET wouldn't be able to find it.

The number of handoffs to look at is bounded, though; the proxy won't just look for everything in the cluster. The setting "request_node_count" in the proxy config controls how many handoffs will be asked, and the default is twice your replica count.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-06-27 03:59:37 -0500

Seen: 74 times

Last updated: Jun 28 '13