all get requests to files 404 after extending volume size in Swift SAIO installation

asked 2016-12-22 05:51:56 -0500

wenyuan gravatar image

updated 2016-12-22 23:37:22 -0500

We have a Swift SAIO installation on one of our servers. It is to store all the user uploaded files from a small internal app we developed. It works all fine until one of our ops tried to extend the swift loop volume this afternoon.

What he did is to stop swift, copy everything to a temp directory, truncate the loop file, remake the xfs, mount and copy files back.

After restarting swift, requests to every file that uploaded before became 404, but you can still list the files. The newly uploaded files are fine.

[root@host-10-20-4-44 swift]# curl -X GET -H "X-Auth-Token: AUTH_tkd6b685264d0846e2acb9cf76dd0515e8" http://10.20.4.44:9999/v1/AUTH_test/test
...
null/1481882937_1041165417646604798_12345/1.png
...
[root@host-10-20-4-44 swift]# curl -X GET -H "X-Auth-Token: AUTH_tkd6b685264d0846e2acb9cf76dd0515e8" http://10.20.4.44:9999/v1/AUTH_test/test/null/1481882937_1041165417646604798_12345/1.png
<html><h1>Not Found</h1><p>The resource could not be found.</p></html>[root@host-10-20-4-44 swift]#

The Swift version is "2.9.1.dev164", os is centos 7, and we followed http://docs.openstack.org/developer/swift/development_saio.html (http://docs.openstack.org/developer/s...) to set up a saio environment.

[root@host-10-20-4-44 swift]# curl http://10.20.4.44:9999/info
{"bulk_delete": {"max_failed_deletes": 1000, "max_deletes_per_request": 10000}, "versioned_writes": {}, "container_quotas": {}, "crossdomain": {}, "swift": {"max_file_size": 5368709122, "account_listing_limit": 10000, "account_autocreate": true, "max_meta_count": 90, "max_meta_value_length": 256, "container_listing_limit": 10000, "extra_header_count": 0, "max_meta_overall_size": 4096, "version": "2.9.1.dev164", "max_meta_name_length": 128, "max_header_size": 8192, "policies": [{"default": true, "name": "gold", "aliases": "gold"}, {"name": "silver", "aliases": "silver"}, {"name": "ec42", "aliases": "ec42"}], "max_object_name_length": 1024, "max_account_name_length": 256, "strict_cors_mode": true, "allow_account_management": true, "max_container_name_length": 256}, "tempurl": {"outgoing_remove_headers": ["x-object-meta-*"], "incoming_remove_headers": ["x-timestamp"], "incoming_allow_headers": [], "methods": ["GET", "HEAD", "PUT", "POST", "DELETE"], "outgoing_allow_headers": ["x-object-meta-public-*"]}, "bulk_upload": {"max_failed_extractions": 1000, "max_containers_per_extraction": 10000}, "container_sync": {"realms": {"SAIO": {"clusters": {"SAIO_ENDPOINT": {"current": true}}}}}, "ratelimit": {"account_ratelimit": 0.0, "max_sleep_time_seconds": 60.0, "container_ratelimits": [], "container_listing_ratelimits": []}, "slo": {"max_manifest_segments": 1000, "min_segment_size": 1, "max_manifest_size": 2097152}, "account_quotas": {}, "staticweb": {}, "tempauth": {"account_acls": true}}

Any idea? Can we recover the files? Thanks!

edit retag flag offensive close merge delete

Comments

swift-get-nodes object-ring.gz account container obj gives you the physical location of an object under normal circumstances; did you try it? You should also check the log; normally the syslog.

As to why it doesn't work: I wonder if the metadata was copied with the files.

Bernd Bausch gravatar imageBernd Bausch ( 2016-12-22 08:24:37 -0500 )edit

@Bernd Bausch Thanks! I found the objects are quarantined http://pastebin.com/hvFzRh40 Is there a way to recover them? We only enabled the log after the incident, but could not find any thing useful...

wenyuan gravatar imagewenyuan ( 2016-12-22 21:22:20 -0500 )edit

My knowledge of swift is somewhat limited. I would guess you could just take the quarantined files and upload them as objects again. Metadata, if lost, is probably lost for good (but who knows?)

Finally a tip: swiftstack.com has good articles on background and practice of Swift.

Bernd Bausch gravatar imageBernd Bausch ( 2016-12-26 03:06:50 -0500 )edit