So, I am seeing some very strange behavior with Openstack Swift. If I start all the swift processes, and say do an operation (eg. upload a file), then if I kill all my Swift processes, copy all the data directories to some other temp location, delete the content of all the data directories, copy all the temp directories back to data dir and start my swift processes again, I am not able to perform a GET for the object. (I can access the container info but swift download fails).

I have spend hours debugging this very strange behavior. Anyone else has experienced this problem ?

Steps for reproducing the bug: (Assuming 4 Data Dirs say in /mnt/sdb1/ dir)

=> swift-init all start => swift upload some_container some_file
=> (Can be followed by any no of swift operations)
=> swift-init all kill
=> cp -R /mnt/sdb1/1 /mnt/sdb1/1_temp (copy from original to a temp dir)
=> cp -R /mnt/sdb1/2 /mnt/sdb1/2_temp (copy from original to a temp dir)
=> cp -R /mnt/sdb1/3 /mnt/sdb1/3_temp (copy from original to a temp dir)
=> cp -R /mnt/sdb1/4 /mnt/sdb1/4_temp (copy from original to a temp dir)
=> rm -rf /mnt/sdb1/1/* (delete all the contents of original data dir)
=> rm -rf /mnt/sdb1/2/* (delete all the contents of original data dir)
=> rm -rf /mnt/sdb1/3/* (delete all the contents of original data dir)
=> rm -rf /mnt/sdb1/4/* (delete all the contents of original data dir)
=> cp -R /mnt/sdb1/1_temp /mnt/sdb1/1 (copy to original from the temp dir)
=> cp -R /mnt/sdb1/2_temp /mnt/sdb1/2 (copy to original from the temp dir)
=> cp -R /mnt/sdb1/3_temp /mnt/sdb1/3 (copy to original from the temp dir)
=> cp -R /mnt/sdb1/4_temp /mnt/sdb1/4 (copy to original from the temp dir)

=> swift-init all start (start swift again)
=> swift list (this succeeds)
=> swift list some_container (this succeeds)
=> swift download some_container some_file (this fails)

This is very strange considering the fact that I just moved the content.

Any pointers ?