Ask Your Question

torgomatic's profile - activity

2015-10-21 07:57:47 -0500 received badge  Good Answer (source)
2015-08-07 11:12:29 -0500 answered a question Is there a way in OpenStack Swift to upload files via pre-signed URL's (as can be done with Amazon S3) ?

Swift has tempurls that can do exactly that.

http://docs.openstack.org/icehouse/co...

I've linked the docs for Icehouse, but it's the same in all versions of Swift.

2015-04-20 18:34:18 -0500 answered a question how can I see the full URL of files saved in Swift?

You can use Swift's tempurl feature for this: http://docs.openstack.org/trunk/confi...

Basically, you hash together the object name, link expiration time, verb (here, "GET"), and an account-level secret and then stick that in a query parameter. That link allows public access until the expiration time.

Hint: If you want a link that's good for basically forever, just make the expiration time really large.

2015-04-20 18:30:39 -0500 answered a question Swift: How do I measure bandwidth (bytes in/out) for a container (not just an object)

Swift does not have support for per-container/per-account bytes-transferred metrics. Ceilometer provides a few, and more are available in StatsD format (see http://docs.openstack.org/developer/s...), but nothing of the granularity you're looking for.

2015-04-18 23:43:43 -0500 answered a question updating/changing reseller prefix?

For each account pair AUTH_xyz and KEY_xyz, you'll need to copy every container and object from AUTH_xyz to KEY_xyz, then delete AUTH_xyz.

Swift uses a hash of the entity name (account, account/container, account/container/object) to determine placement of the entity on disk. Renaming an account thus changes the hash of everything therein, and so requires a copy-and-delete of everything in the account.

Note that though I've said "rename", Swift's API has no such operation. To "rename" something is really to copy it and then delete the original.

Side note: from the standpoint of getting your cluster working again, the fastest way is to put the reseller prefix back. It's just an arbitrary string. Even if you're running multiple auth systems, just ensure their prefixes don't collide; that's all.

2015-04-16 20:15:14 -0500 received badge  Nice Answer (source)
2015-04-15 22:46:00 -0500 answered a question What is the reason to send writes to all replica nodes?

When a client performs an object PUT and Swift returns a 201 response, that means that the object has been safely stored and will be retrievable in the future.

If Swift were to write to only one disk and that disk failed before a second replica was written, then the object would be lost.

Disks fail at a high enough rate that writing to one disk is not enough to be safe.

2015-04-13 13:48:45 -0500 answered a question State of swift after ring rebalance

Swift will automatically load and begin use new rings in a very short time after they're pushed; on the order of 15 seconds. It's not quite instantaneous, but it's close enough you can treat it that way.

2015-04-08 17:01:10 -0500 answered a question how the migration of data in swift is done?

Basically, the object replicator scans its local disks to find partitions, pushes the data in those partitions to the primary nodes that it belongs on, and if the partition does not belong on the local node, deletes it. Deletion will only occur if it was successfully pushed to all the primary nodes, though, so if one is down then durability is not compromised.

There's no coordination between instances of the object replicator, so it's entirely possible for two replicators to be pushing the same partition to the same node at the same time. If this happens, then some bytes are transferred needlessly, but it doesn't hurt anything.

2015-03-15 11:40:14 -0500 answered a question Swift read problems

Odds are that your /etc/swift/swift.conf is not the same across all your nodes, in particular the entries swift_hash_path_prefix and swift_hash_path_suffix. If those don't match, then the object auditor will compute a different hash than the proxy will for a given object name, and so it will believe the object corrupted.

Always keep /etc/swift/swift.conf the same across all the nodes in your cluster.

2015-02-20 23:40:32 -0500 answered a question Encrypt swift objects

Server-side encryption is not currently implemented in Swift. There are some efforts underway, but nothing has yet made it to a release. Any encryption done at this time will have to be done client-side.

2015-02-15 12:07:07 -0500 answered a question Necessary to mount a partition for swift ?

You can set mount_check = false in the various Swift config files to let you use a plain old directory that isn't a mount point.

The usual reason for setting that is to avoid scribbling objects onto the root filesystem. If you've got, say, a 1TB root and 48 4TB data disks, then you'll definitely want mount_check = true (the default). Otherwise, you get one or two disks unmounted and then replication fills up your root filesystem.

2015-02-04 13:22:47 -0500 answered a question How to create symlinks in swift?

The approach that you mentioned using manifests[1] is the way to do it. If you are using a newer Swift, I'd recommend using static large objects[2] instead of dynamic large objects. The concept is similar, but the creation method differs slightly. However, static large objects give you assurance that the referenced object has not changed out from under you.

[1] https://developer.rackspace.com/blog/...

[2] http://docs.openstack.org/api/opensta...

2014-11-08 17:58:13 -0500 answered a question swift capabilities 401 Unauthorized Authentication required

Sounds like you're using Keystone for auth, but you haven't set delay_auth_decision = true in the authtoken middleware's config.

2014-10-21 22:58:36 -0500 received badge  Nice Answer (source)
2014-10-20 19:24:17 -0500 received badge  Nice Answer (source)
2014-10-20 17:20:55 -0500 answered a question Can I force an NTP sync in Swift hosts?

If the time is only off by a couple of hours, just fix it.

If it's off by more, look at the replicators' reclaim_age settings. If the time difference is near reclaim_age, then it's possible that deletions won't get properly replicated before the tombstones are cleaned up. To handle that, increase reclaim_age by the time difference, restart the Swift daemons, and then fix the time. After the original reclaim_age has passed, you can lower reclaim_age back to its starting value.

2014-10-20 17:16:33 -0500 answered a question swift account reaper with tempauth

If you want a Swift account to be deleted, issue a DELETE request to the account. This request will almost certainly require a reseller-admin token, though that depends on which auth system your cluster has.

2014-10-20 17:15:13 -0500 answered a question swift and amazon as ring pairs

Swift does not store data in other object storage systems. Any data that requires mirroring in S3 or Azure will have to be mirrored outside of Swift.

2014-09-18 18:08:55 -0500 answered a question How to get a report of swift disk usage by user

You can get an account's usage with a HEAD request. However, Swift does not maintain any central list of accounts, so you'll have to ask the authentication system for that.

2014-09-11 13:22:10 -0500 answered a question Swift - replication issue

When Swift deletes an object, it also deletes its entry from the partition's hashes.pkl file. Replication uses hashes.pkl to avoid doing a bunch of disk IO for an up-to-date partition.

You deleted the file by going around Swift and talking to the filesystem directly, so replication thinks the file is still there.

In an actively-used Swift cluster, other object activity (PUTs, DELETEs) will cause hashes.pkl to get updated, and after that happens, replication will restore the manually-deleted object. However, that won't happen until there's other activity that coincidentally fixes hashes.pkl, so it may take a while. On an inactive demo system, "a while" translates to "forever", hence what you saw.

2014-07-14 14:49:04 -0500 answered a question tempURL key generation and how to block it.

No, there's no way to disable that. The tempurl key is just a piece of account metadata with a particular name, and Swift does not offer a way to disable editing of account metadata.

2014-07-14 14:47:18 -0500 edited question tempURL key generation and how to block it.

I have a server that can hand out temp URLs to access objects in the object store.

A potential problem is that if someone were to generate a new tempURL key it would invalidate the one used by the server and neither the person who did this or the server would be aware until you have clients complaining that the URLs do not work.

Is there any way to disable the tempURL key generation so the key can not be changed so easily? It is the type of thing that a developer might do unaware of the side effects.

2014-07-07 03:32:39 -0500 received badge  Nice Answer (source)
2014-05-14 01:57:12 -0500 received badge  Nice Answer (source)
2014-05-13 18:00:30 -0500 answered a question swift: search objects in container by name

You can search by prefix with the prefix query parameter; setting prefix=abc will retrieve only entries for objects starting with "abc".

2014-05-13 17:48:33 -0500 answered a question swift: meta data of all objects in a container

Swift does not support that. The only things stored in the container DB, and hence the only things available in the container listing, are what you see above.

2014-04-28 06:33:29 -0500 received badge  Nice Answer (source)
2014-03-25 13:57:20 -0500 answered a question Container data after deletion of account

The deletion of everything under a deleted account is the responsibility of the account reaper. Ensure that it is running on all your storage nodes that have accounts on them.

2014-03-24 13:08:45 -0500 commented answer Running Swift with SAN backend

Note that you have to be careful to tell Swift about the actual failure domains. If you have many disks in different zones all backed by one SAN thingy, then a single hardware failure can cause unavailability or even data loss.

2014-03-18 12:35:37 -0500 edited question push/rebalance/repush this ring

When I added a new zone to my existing swift cluster with

swift-ring-builder account.builder add r1z4-192.168.1.22:6002/sdb1 100
swift-ring-builder container.builder add r1z4-192.168.1.22:6001/sdb1 100
swift-ring-builder object.builder add r1z4-192.168.1.22:6000/sdb1 100

then I did

swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance

I got the following messages:

Reassigned 1024 (100%) partitions. Balance is now 2193
Note: Balance of 2193 indicates you should push this ring, wait at least 1 hour, and rebalance/repush

what does the above mean?

Does mean I have to first copy the ring to the all the storage nodes including the new one 192.168.1.22 first, and wait for one hour, and copy the new ring to all the nodes again? then rebalance and then copy the new ring files to

2014-03-17 18:57:03 -0500 answered a question How to create unique object names?

If you want unique names, you have to make sure the names you generate are unique. Swift won't help you here, as an object PUT will cheerfully overwrite an object of the same name.

In this case, it looks like you could use timestamps with microsecond or nanosecond precision (whatever you've got laying around) to give you ordering, plus a UUID to give you uniqueness in the rare case where two tests start on the same microsecond.

2014-03-17 03:45:51 -0500 received badge  Nice Answer (source)
2014-03-17 03:44:19 -0500 received badge  Nice Answer (source)
2014-03-11 18:17:18 -0500 answered a question does push of rings require reloading servers?

The Swift processes will pick up ring changes without a reload, so the scp is all you need to do.

2014-03-10 12:33:47 -0500 answered a question Could someone tell me the best practice to set the min part hour in swift?

Find your replicators' cycle time by looking in the logs for "replication complete" and round that up to the next hour.

min_part_hours is intended to keep your data available even while data is moving after a rebalance, so it needs to be long enough for the replicators to actually copy all that data across the network to its new homes. How long that takes depends a whole lot on the particular cluster, but you can find out by looking in the object replicator logs.

2014-03-07 17:00:48 -0500 answered a question How to minimize the chance that a file can't be read during Swift rebalance process?

That's really close, but not quite there.

Each object has three primary nodes with copies of your data on them. For a GET request, the proxy tries all three, and then tries the first three handoffs. During a rebalance, swift-ring-builder reassigns at most one of the primary nodes, leaving at least two replicas unchanged and hence available. The other nodes won't get moved until at least min_part_hours hours have passed, so as long as min_part_hours is big enough, your data will always be available even after a rebalance

Note: with N replicas, it'll only move 1 and keep N-1 in place, so if you're only running with 1 replica, you will suffer unavailability after a rebalance.

2014-02-24 19:25:32 -0500 answered a question Where does Havana Swift define region settings?

Devices are placed in regions when an admin adds them to the ring with swift-ring-builder. There's no other configuration necessary.

As for optional configuration, see read_affinity and write_affinity in the docs: http://docs.openstack.org/developer/s... . If you have a multi-region cluster, then read_affinity is very likely to be something you want to set up. On the other hand, write_affinity requires careful thought before using, so don't rush in there.

2014-02-24 19:18:54 -0500 answered a question how can we create a dropbox kind of mechanism in open stack swift?

Access control in Swift is governed by the auth system, which is pluggable. Sounds like you'd need to write your own auth system to do what you want.

http://docs.openstack.org/developer/s...

2014-02-24 19:13:51 -0500 answered a question Offline Swift ring.gz files

Yes, that'll work. In fact, the only purpose of the builder files is to produce the ring.gz files, so you don't need to have the builders on the proxy machines at all.

2014-02-20 19:53:27 -0500 answered a question Swift - encryption road map

Right now, it's all up to the client.

There may or may not be some future plans for something someday in Swift, but don't wait for it.

Besides, if you really want to keep your data secure, you have to encrypt it before it leaves your machine. If you rely on a service provider to encrypt your data for you, you're trusting them not to copy the plaintext somewhere (right into the NSA's waiting hands) before encrypting it, and of course if you can get the data back out in its original plaintext form, that means your service provider can just decrypt your data and hand it over to $adversary any time they want.

2014-02-11 07:49:29 -0500 received badge  Good Answer (source)
2014-02-11 07:49:29 -0500 received badge  Enlightened (source)
2014-02-03 14:06:01 -0500 answered a question can i use ssd caching with swift object storage?

Might work. It looks like bcache does not cache sequential reads and writes on the fast device, so the object auditor shouldn't thrash the cache too badly. Might get you caching of all the filesystem metadata, which would probably boost the speed some. On the other hand, bcache does not cache sequential reads, so if you're hoping for hot objects to be cached on the SSD, it doesn't look like that'll happen.

Ultimately, Swift just wants a mounted filesystem somewhere to write data into, so it will work in the sense that Swift API operations will complete correctly; as for performance, try it and see.

Report back and let us know how it goes. :)

2014-02-03 13:41:57 -0500 answered a question swift essex version extending authentication

Yes, it worked like that back in Essex release too, and for some time even before that. If you want custom auth stuff, all you need to do is write your own middleware and drop it in the pipeline.

2014-02-03 13:40:31 -0500 edited question swift essex version extending authentication

Hi everyone,

While reading documents of the new version(grizzly) of swift i came acroos the following section about extending authentication here and here.Currently we cannot afford to change version of swift (essex) since it deployment in both our test and product environments.

I could not find wiki pages for the essex version, so i have to ask here. Does essex version support this feature in the same way?

Thanks

2014-01-30 13:16:09 -0500 received badge  Nice Answer (source)
2014-01-28 18:50:22 -0500 answered a question Order of sub-objects in a large object...

The object segments (sub-objects) are fetched in the same order they appear in the container listing, which is plain old bytewise-lexical order.

If you want to list just the segments of a dynamic large object, get the X-Object-Manifest value from the manifest and use it as the prefix in your container listing (more details on prefix query param at http://docs.openstack.org/api/openstack-object-storage/1.0/content/list-objects.html).

2014-01-28 18:45:37 -0500 answered a question [Swift] Can I query objects based on their metadata?

At this time, Swift does not support object lookups by anything except name.