Mitaka - Object Storage - Invalid response 405 from

asked 2016-08-25 05:23:59 -0500

anonymous user

Anonymous

Hi guys, i need a little help to understand what's happening.

I have an Object Storage infrastructure and few day ago i had a disk fault. I have changed the disk, remounted, set permission and so on.

Using swift-dispersion-report on old populated data I received this:

Queried 51 containers for dispersion reporting, 0s, 0 retries 100.00% of container copies found (102 of 102) 
Sample represents 4.98%  of the  container partition space 
Queried 51 objects for dispersion reporting, 0s, 0 retries There were 37 partitions missing 0 copies. ! There were 14 partitions missing 1 copy.
 86.27% of object copies found (88 of 102) Sample represents 4.98% of the object partition space

Seems that was an issue related to object dispersion. Saw this, I thought that may be was an issue on the object-replication process...looking in the logs file i found this error:

NODE-1 object_replication: Starting object replication pass.
NODE-1 object_replication: Invalid response 405 from 192.168.0.3
NODE-1 object_replication: message repeated 452 times:  [ Invalid response 405 from 192.168.0.3]
NODE-1 object_provision: Starting object replication pass.
NODE-1 object_provision: Can't find itself 192.168.87.2 with port None in ring file, not replicating
NODE-1 object_provision: Nothing replicated for 0.00172591209412 seconds
NODE-1 object_provision: Object replication complete. (0.00 minutes)
NODE-1 object_replication: Invalid response 405 from 192.168.0.3
NODE-1 object_replication: message repeated 469 times: [ Invalid response 405 from 192.168.0.3]
NODE-1 object_replication: 923/923 (100.00%) partitions replicated in 2.65s (348.34/sec, 0s remaining)
NODE-1 object_replication: 0 successes, 973 failures
NODE-1 object_replication: 22415 suffixes checked - 0.00% hashed, 0.00% synced<
NODE-1 object_replication: Partition times: max 0.0047s, min 0.0017s, med 0.0025s
NODE-1 object_replication: Object replication complete. (0.04 minutes)

Now, I can't understand what's happening so i executed swift-recon with this output:

$ swift-recon --all 

===============================================================================
Starting reconnaissance on 10 hosts
===============================================================================
[2016-08-25 09:57:28] Checking async pendings
[async_pending] - No hosts returned valid data.
===============================================================================
[2016-08-25 09:57:28] Checking auditor stats
[ALL_audit_time_last_path] low: 0, high: 3702, avg: 1851.2, total: 18512, Failed: 0.0%, no_result: 0, reported: 10
[ALL_quarantined_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 10
[ALL_errors_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 10
[ALL_passes_last_path] low: 1, high: 3983, avg: 1992.0, total: 19920, Failed: 0.0%, no_result: 0, reported: 10
[ALL_bytes_processed_last_path] low: 0, high: 36024275459, avg: 18012137729.5, total: 180121377295, Failed: 0.0%, no_result: 0, reported: 10
[ZBF_audit_time_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 10
[ZBF_quarantined_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 10
[ZBF_errors_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 10
[ZBF_bytes_processed_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 10
===============================================================================
[2016-08-25 09:57:28] Checking updater times
[updater_last_sweep] low: 0 ...
(more)
edit retag flag offensive close merge delete