Ask Your Question

tnewsome's profile - activity

2016-03-10 10:47:04 -0500 received badge  Famous Question (source)
2016-02-08 09:17:25 -0500 received badge  Famous Question (source)
2015-01-20 17:40:08 -0500 received badge  Notable Question (source)
2015-01-20 17:40:08 -0500 received badge  Popular Question (source)
2014-11-17 21:21:25 -0500 received badge  Popular Question (source)
2014-11-17 21:21:25 -0500 received badge  Notable Question (source)
2013-10-17 21:54:00 -0500 answered a question Why does my cluster see so many errors?

My client sees these errors as the connection being closed before all data is received.

I don't understand what we would find out by adjusting node_timeout.

There are all kinds of errors. :-P I've pasted some recent ones below. The logs I pasted are for all servers. You can see immediately after the timestamp it shows the hostname of the server the data came from.

I've doubled the number of object workers from 8 to 16. I'll see if that reduces errors noticeably.

Here are some errors: swift105: Oct 17 14:24:32 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: txe1754390285a49c38cbd128aecaa5fe7) swift105: Oct 17 14:24:32 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d4 (saving for async update later): Timeout (3s) (txn: txf6c8946ffe464ee082891185f8d9d189) swift105: Oct 17 14:25:03 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d3 (saving for async update later): ConnectionTimeout (0.5s) (txn: txa90447e0ab4044ea95a9c0d03a0c53aa) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: tx10e193ec5c9c44a08a1918580b789d57) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: tx2695ba920505408eb91dd23faaeee349) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: tx51fdefef4dc744d99461d579e37a8d43) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: txd50e07b5f449486a884c24bf7852cf26) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: txed7ab47336ef4900b3bd4eb98598917f) swift105: Oct 17 14:25:40 swift105 object-server ERROR container update failed with 172.17.10.73:6001/d4 (saving for async update later): Timeout (3s) (txn: txfed5461a2d644faa9b4c3aac5a3ddce5) swift105: Oct 17 14:26:35 swift105 object-server ERROR container update failed with 172.17.10.74:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx3731beb2dedf46eabe748bae78e30c85) swift105: Oct 17 14:26:35 swift105 swift ERROR with Object server 172.17.10.73:6000/d2 re: Trying to DELETE /.expiring_objects/1381536000/1381541992-AUTH_software/rpmcache/afe96635c9f7f8372525a970511a64ba5eb0702f14a4664dc240045c7308a260.data: ConnectionTimeout (0.5s) (txn: txa7a32e4bafd44d83b23451a14c05bbd1) swift105: Oct 17 14:26:35 swift105 swift ERROR with Object server 172.17.10.75:6000/d4 re: Trying to DELETE /.expiring_objects/1381536000/1381541992-AUTH_software/rpmcache/afe96635c9f7f8372525a970511a64ba5eb0702f14a4664dc240045c7308a260.data: ConnectionTimeout (0.5s) (txn: txa7a32e4bafd44d83b23451a14c05bbd1) swift105: Oct 17 14:32:53 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx3a05ad050e124af4a56fa74b56b1e5df) swift105: Oct 17 14:32:53 swift105 object-server ERROR container update failed with 172.17.10.72:6001/d2 (saving for async update later): ConnectionTimeout (0.5s) (txn: tx8383669ad0ba44e3a155b1664b03840c) swift105: Oct 17 ... (more)

2013-10-14 16:26:04 -0500 answered a question Why does my cluster see so many errors?

The specific error that I'm talking about is when my client application does a GET, but the connection gets closed before all bytes are transferred. This looks as follows in the swift logs: Oct 14 02:51:13 swift104 object-server 172.17.10.73 - - [14/Oct/2013:09:51:13 +0000] "GET /d4/201683/AUTH_software/buildcache/76b67f599086944d4ee4eb14b9391b9d2023aa9f.data" 200 10270720 "-" "txe41b50d1e237489a862b1e7bd59daf8c" "-" 0.0005 Oct 14 02:51:30 swift102 proxy-server 172.22.34.14 172.22.34.14 14/Oct/2013/09/51/30 GET /v1/AUTH_software/buildcache/76b67f599086944d4ee4eb14b9391b9d2023aa9f.data HTTP/1.0 200 - - software%2CAUTH_tk62d08be9f72543fba543d5dbe673c1e7 - 6553600 - txe41b50d1e237489a862b1e7bd59daf8c - 16.5518 - Oct 14 02:51:30 swift102 proxy-server ERROR with Object server 172.17.10.75:6000/d4 re: Trying to read during GET: ChunkReadTimeout (10s) (txn: txe41b50d1e237489a862b1e7bd59daf8c) (client_ip: 172.22.34.14)

Oct 14 06:11:22 swift102 object-server 172.17.10.74 - - [14/Oct/2013:13:11:22 +0000] "GET /d4/206078/AUTH_software/buildcache/0bb1b59f6f177ceace00c0bcdf56eca221b06cb5.data" 200 4783308800 "-" "tx1d86b18d378f4b94a2caadaccb70fadd" "-" 0.0584 Oct 14 06:11:32 swift103 proxy-server 172.22.34.14 172.22.34.14 14/Oct/2013/13/11/32 GET /v1/AUTH_software/buildcache/0bb1b59f6f177ceace00c0bcdf56eca221b06cb5.data HTTP/1.0 200 - - software%2CAUTH_tk62d08be9f72543fba543d5dbe673c1e7 - 65536 - tx1d86b18d378f4b94a2caadaccb70fadd - 10.5687 - Oct 14 06:11:32 swift103 proxy-server ERROR with Object server 172.17.10.73:6000/d4 re: Trying to read during GET: ChunkReadTimeout (10s) (txn: tx1d86b18d378f4b94a2caadaccb70fadd) (client_ip: 172.22.34.14)

I don't remember playing with ThreadPools. Given that my problem is disk I/O bandwidth, I don't think it makes a difference how those accesses get in the queue (except that it would help to not audit each disk in turn). We're running swift 1.4.0, with the following configuration (on swift100): account-server.conf:[DEFAULT] account-server.conf:workers = 4 account-server.conf:log_statsd_host = 172.17.23.35 account-server.conf:log_statsd_port = 8125 account-server.conf:log_statsd_default_sample_rate = 0.5 account-server.conf:[pipeline:main] account-server.conf:pipeline = recon account-server account-server.conf:[app:account-server] account-server.conf:use = egg:swift#account account-server.conf:[account-replicator] account-server.conf:[account-auditor] account-server.conf:[account-reaper] account-server.conf:[filter:recon] account-server.conf:use = egg:swift#recon account-server.conf:recon_cache_path = /var/cache/swift container-server.conf:[DEFAULT] container-server.conf:workers = 8 container-server.conf:[pipeline:main] container-server.conf:pipeline = recon container-server container-server.conf:[app:container-server] container-server.conf:use = egg:swift#container container-server.conf:[container-replicator] container-server.conf:[container-updater] container-server.conf:[container-auditor] container-server.conf:[container-sync] container-server.conf:[filter:recon] container-server.conf:use = egg:swift#recon container-server.conf:recon_cache_path = /var/cache/swift object-expirer.conf:[DEFAULT] object-expirer.conf:swift_dir = /etc/swift object-expirer.conf:user = swift object-expirer.conf:log_statsd_host = 172.17.23.35 object-expirer.conf:log_statsd_port = 8125 object-expirer.conf:log_statsd_default_sample_rate = 1 object-expirer.conf:[object-expirer] object-expirer.conf:interval = 300 object-expirer.conf:auto_create_account_prefix = . object-expirer.conf:report_interval = 300 object-expirer.conf:concurrency = 1 object-expirer.conf:processes = 6 object-expirer.conf:process = 0 object-expirer.conf:[pipeline:main] object-expirer.conf:pipeline = catch_errors cache proxy-server object-expirer.conf:[app:proxy-server] object-expirer.conf:use = egg:swift#proxy object-expirer.conf:[filter:cache] object-expirer.conf:use = egg:swift#memcache object-expirer.conf:[filter:catch_errors] object-expirer.conf:use = egg ... (more)

2013-10-11 00:34:46 -0500 answered a question Why does my cluster see so many errors?

So I put some more load on the cluster, and am back to one failure every two hours.

What's the general recommendation on retries? These failures happen infrequently enough that I bet I can just retry and it'll work. The extra delay is not a big problem for my application. Is this a good idea?

2013-10-07 21:48:14 -0500 answered a question Why does my cluster see so many errors?

Right. We went with the dumb approach, and run every swift process (including the proxy) on every server. The only exception is the object-expirer, which we only run on a single server. Each of 6 servers has a single E31270 (3.4GHz, 8 cores including hyperthreading), 16GB RAM, 4 x 4TB disk. The servers never swap. We're using isize=1024 with a patched Linux kernel 3.4.43 x86_64. (I believe all our patches are network related and shouldn't affect disk I/O.)

2013-10-07 19:16:15 -0500 answered a question Why does my cluster see so many errors?

OK, deadline scheduler makes a big improvement. With that I don't see any errors, as long as I keep files_per_second and zero_bytes_files_per_second at 1. At 10 I'm seeing one client-visible error every 10 hours. I'm experimenting with 5.

2013-10-03 22:24:44 -0500 answered a question Why does my cluster see so many errors?

Thanks for the zero_byte_files_per_second pointer. I missed that. Setting it to 1 (which is effectively turning it off), I still get 4 client-visible errors per day, where a GET fails.

We were just using the cfq scheduler. I switched it to the deadline scheduler. I'll report back what happens.

I'm still a bit concerned that so much tuning is required for what should be a light load on the overall cluster.

2013-09-30 15:08:52 -0500 answered a question Why does my cluster see so many errors?

OK, I spent last week doing some expirements. Turing off all background consistency processes was a good suggestion.

There are 2 processes which have enough impact to cause periodic failures: 1. object-auditor. After throttling it back a lot (1MB/s, 1 file/s) it still causes about 1 problem every 18 hours. Looking at the IO graph it still periodically hits the disks heavy. I suspect this is the ZBF audit which doesn't appear to be affected by the files_per_second configuration. 2. object-replicator. With everything except the object-auditor running, I see about 1 problem every 2 hours. I don't think there's an option to throttle it.

Both these processes seem to hit each disk in succession. Ie. they'll hit sda for a while, then sdb, then sdc, and so on. Their impact would be a lot less if they switched disks more often. Their cache hits on the file metadata would also be less, though. On the other hand, overall audit/replicate performance might be larger if they distribute their load over each disk, because presumably they're spending most of their time waiting on IO.

2013-09-19 17:54:10 -0500 answered a question Why does my cluster see so many errors?

I ran some more experiments. Specifically: 1. Running without any object expirers: No difference. 2. Disabling the POST updates in my application: No difference.

It looks like the PUT/GET/HEAD load is just too high, but the numbers are so low that I have a hard time believing that. Average object size is 26MB. Is that causing my trouble? Some objects (1 in 1000 or so) are 2GB in size.

Is there any place where I can find a list of swift cluster configurations along with the workload they support?

2013-09-18 16:13:32 -0500 answered a question Why does my cluster see so many errors?

I've confirmed that POSTs on a 2GB file takes about .25s (using the swift CLI client, so including time to authenticate). So it looks like fast-POST is enabled.

I don't update delete-at with every request (anymore). There is less than one of these per second (to the proxy server). For reference, the proxy servers see a combined 0.5 PUTs/second, 0.5 POSTs/second, 2 HEADs/second, 4 GETs/second. Which really doesn't seem that high.

Increasing the number of workers has significantly increased the disk business, and with it the number of errors swift reports.

I did discover we were running an object expirer on each of 6 servers (without configuring them to each expire their own part). I turned off 5 of them when I increased the workers, but the net result is still more disk business and errors. Just now I turned off the last object expirer, just to see if that improves things. (So far it doesn't look like it, but let's give it some time.)

All data is using the same storage, etc. I'm not really against distributing things smarter, but I'm still baffled that it would be required. This exact functionality had no problems with an 8-disk RAID10. Why does my 24-disk swift cluster have so much trouble keeping up?

I'll dig into some more of your suggestions today. I really appreciate all the help you're giving me.

2013-09-17 18:03:49 -0500 answered a question Why does my cluster see so many errors?

I am using POST to update the delete-at time. I do have object_post_as_copy set false. (At least it's there in the config file. I didn't see a way to confirm that it's actually happening.) From your reaction it sounds like a weird thing to do. What is the usual solution to a problem like this? Store delete-at in some other db, and write my own object expirer?

I have read the general service tuning section, but it doesn't give much guidance. I'm running the object-server with 8 workers. Presumably that's enough to avoid problems caused by one object server being temporarily hung. The CPU on each server is only 50% used. Interestingly it's the container server/replicator that are consistently at the top of top output. Fairly often I see a container-replicator using 100% of one core. Is there any downside to having too many workers? I'm thinking of just giving each server 16 workers, with a concurrency of 1, and let Linux sort it out.

Does the object expirer log a delete also when an old copy of metadata is deleted? If so, that would explain the large number of deletes, and those files should be tiny so deleting them shouldn't add significant load to anything.

2013-09-17 00:04:10 -0500 asked a question Why does my cluster see so many errors?

I've converted an object cache from using local disk (8 drives, RAID10) to swift (6 servers, 4 drives per server). I'm getting significant amount of swift errors, to the point where several times a day my clients fail to read an object. They'll get part of the file, but then swift gets a 10s ChunkReadTimeout and the transfer is halted.

Looking at the server metrics in OpenTSDB, these errors always occur when a bunch of disks all show very high iostat.disk.msec_total. So my assumption is that something is happening on swift that's hammering the disks. But I don't understand why swift is putting such a heavy load on them.

One thing that might be an unusual swift use case, is that this cache is supposed to be LRU. To make that happen, every time there's a read from the cache and the entry will expire soonish, we update the delete-at time. This happens about 15 times per minute.

This swift cluster only contains a single active container (about 4TB, 500,000 objects) with another 2TB, 150,000 objects in another container that was used for testing and is now just sitting there.

Combining the logs from all the swift servers, I see: DELETE: 20/second (container, object) HEAD: 4/second (container, object, account) GET: 7/second (container, object, account) PUT: 5/second (container, object, account)

Why are there so many deletes? Is the load I'm putting on the cluster just more than swift can handle?

2013-02-28 22:37:59 -0500 answered a question Can Swift serve up a 301/303 redirect?

Thanks for pointing me towards middleware. That looks like the right solution. Disregard my question about getting it included in core swift. With a flexible system like this there's no need!

2013-02-28 22:37:18 -0500 answered a question Can Swift serve up a 301/303 redirect?

Thanks Samuel Merritt, that solved my question.

2013-02-28 21:39:18 -0500 answered a question Can Swift serve up a 301/303 redirect?

I don't think that does what I want. I really do want a 3xx redirect. The issue is that I have clients fetching the same file through a lot of different URLs. So on the server I'll have (with simplified paths) .../project1/baseFile .../project2/baseFile .../project3/baseFile .../canonical/baseFile

I'll have clients getting the baseFile for project[123], and they're really all the same file. These clients are using a standard caching HTTP proxy. Without the redirect, the proxy will end up downloading and keeping 3 identical copies of baseFile in its cache. If project[123]/baseFile redirect to canonical, however, then the proxy will cache the fact that those are redirects (assuming 301), and only download and store the single canonical/baseFile.

I have the same problem in reverse. Many clients are adding files to this database. Each client can quickly check whether a canonical version exists. (This works by putting a hash of the file contents into the name of the canonical file.) Only if it does not, does the client need to upload the canonical version. Once that's done it's quick enough to add a 'redirect' in the appropriate project directory.

Currently I have this setup working with Apache on a single server, but I'd like to use Swift for the reliability and scaling features it provides.

I could try and fix all the clients to not need the redirects, but I don't have as much control over the clients as I do over the server. Another approach is to build something that is a front-end for Swift that does the what I want, but it feels like adding the functionality to Swift directly would not be much extra work, and provide a much better long-term solution.

2013-02-28 20:44:51 -0500 asked a question Can Swift serve up a 301/303 redirect?

Can I upload a redirect to Swift?

The problem I have is that I have a data store of a lot of files, many of which are identical. When a client requests a file, I would like to redirect that client to the canonical URL for that file (also stored in Swift). This allows clients to maximize their cache hit rate, and prevents them from storing multiple copies of the same thing in their cache.

As a quick example, I'd like to do something like: curl –X PUT -i \ -T JingleRocky.jpg \ https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg (https://storage.swiftdrive.com/v1/CF_...) curl –X PUT -i \ -T /dev/null \ -H "X-Object-Meta-Redirect-303: https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg%22 (https://storage.swiftdrive.com/v1/CF_...) \ https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg (https://storage.swiftdrive.com/v1/CF_...)

Then when I GET https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg (https://storage.swiftdrive.com/v1/CF_...) , Swift would return an HTTP 303 return code, with a redirect to https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg (https://storage.swiftdrive.com/v1/CF_...) .

Is this possible? If not, how much work would you estimate this is to implement by an experienced python programmer? Would such a change be considered for inclusion into the main project?

Thank you, Tim