Swift upgrade - creation object OK but POST metadata shows 404 immediatly after - still the object GET a 200

asked 2020-07-28 08:14:05 -0600

locklock gravatar image

updated 2020-07-28 09:55:02 -0600

Hi there, We are looking for some help regarding our current upgrade for our production swift clusters.

Current release : USSURI Previous : Kilo

We were in the process of upgrading from Kilo to Ussuri. But immediatly after we've made the upgrade on our proxy servers, our functionnal tests done by our monitoring system showed failures when trying to create an object and POST metadata on it immediatly after the creation of the object. Still, when we do a GET on the object after the 404, we can see the metadata X-Deleter-After has been successfully added. We also had errors regarding timestamps (in the far future) for the X-Delete-After metadata so we decided to go for Ussuri instead of Stein version. But same result regarding the 404.

We tried to modify some configuration on the storage regarding object/containers updater or expirers , but it looks like the error could come from the proxy servers. The error showed up only after the proxy servers were upgraded to the latest version. The ones which were still in kilo version, were not producing the error when the monitoring system was doing the test through these.

Our current test is doing the following :

1- check keystone & get token 2 - upload an object 3 - Get the object & content 4 - Add metadata x-deleter-after 30s with POST 5 Check file has been successfully deleted after the expire

What we are seing : Token OK. Upload OK. GET object OK. Add Metdata -> 404 'reason' => 'Not Found', 'protocol' => 'HTTP/1.1', 'content' => '<html>

Not Found

The resource could not be found.

</html>', Check Metadata : OK

We are wondering if we did not miss something in the configuration with the upgrade or if there are some new bugs we did not see in the bug currently opened.

Update: We can see by checking the logs when asking a proxy on one site, that the POST is returning a 202 on the local storage but 404 on the 2 others which are on a different geographic site. That could explain the 404 returned, but why is it behaving like this ?

Do you have any idead what could go wrong at the moment ?

Thanks a lot.

edit retag flag offensive close merge delete