Ask Your Question
0

Slow response to expiration time update

asked 2013-07-15 19:06:18 -0600

jcburke gravatar image

Hi, I am using Swift as a cache for files that get up into the 10s of MB, and after fetching one I update its X-Delete-After time (in order to do approximate LRU by having unused stuff expire). I've noticed that for big files, this operation takes longer, which doesn't make sense since I am not sending any data, just headers, and I'm only getting back a small message, and on the server side this is (supposedly) handled asynchronously anyway. I don't have many files on swift since I am basically testing it out right now, so I don't think that is the problem. For example, updating the expiration time for a ~50MB file took min 2.21s, avg 7.5s, and max 28s (!). Here is the command I called 100 times to test this: curl -silent -k -X POST -H 'X-Auth-Token: [my auth token]' -H 'X-Delete-After: 604800' [my storage url]/[container]/[filename]

Any help would be greatly appreciated, taking 7 seconds on average to do this is really not acceptable for our use case (also, for the record, ping times to the servers in question are sub-ms and they have gigabit connections, so I don't think its a network problem).

edit retag flag offensive close merge delete

7 answers

Sort by » oldest newest most voted
0

answered 2013-07-15 21:29:35 -0600

jcburke gravatar image

Thanks David Goetz, that solved my question.

edit flag offensive delete link more
0

answered 2013-07-15 21:31:23 -0600

jcburke gravatar image

Also, thanks for responding so quickly! You guys are clearly doing a great job staying on top of this forum

edit flag offensive delete link more
0

answered 2013-07-15 19:33:36 -0600

david-goetz gravatar image

oh- looks like you also can't update Content-Type with a POST if you have 'object_post_as_copy = false' . Hopefully thats not a problem...

edit flag offensive delete link more
0

answered 2013-07-15 19:21:15 -0600

david-goetz gravatar image

By default all POSTs in swift actually turn into PUT/COPYs. This means that the larger your file is the longer the POST will take because it has to COPY the entire object. If this is not acceptable you can set this flag in the proxy-server.conf:

object_post_as_copy = false

(from docs in etc/proxy-server.conf-sample)

Set object_post_as_copy = false to turn on fast posts where only the metadata

changes are stored anew and the original data file is kept in place. This

makes for quicker posts; but since the container metadata isn't updated in

this mode, features like container sync won't be able to sync posts.

object_post_as_copy = true

edit flag offensive delete link more
0

answered 2013-07-15 20:19:49 -0600

jcburke gravatar image

Thanks David Goetz, that solved my question.

edit flag offensive delete link more
0

answered 2013-07-15 20:34:59 -0600

jcburke gravatar image

Do you think you could explain a little more why this is the default, and what the consequences are of setting it to 'false'? It seems strange to copy the entire object every time its metadata is updated by default, and container-to-container sync seems like a rather obscure feature to be worth such a performance-costly default setting. Is there some other functionality that would break if I change this setting? Also, can it be set on a per-container basis, or does it have to apply to the whole cluster?

edit flag offensive delete link more
0

answered 2013-07-15 21:21:12 -0600

david-goetz gravatar image

Both object_post_as_copy true and false settings are supported by the API so other than the mentioned exceptions POSTs will act as expected. As to why this this setting is needed it has to do with how swift stores its object metadata in the files on disk in conjunction with the metadata stored in the container servers. POST performance is not typically a large concern in swift clusters so the default is set for keeping content-type in sync and the having the ability to use container-sync.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-07-15 19:06:18 -0600

Seen: 47 times

Last updated: Jul 15 '13