Object container error

asked 2014-10-30 11:36:35 -0500

AmitA gravatar image

Hi all,

So I have whats seems to be a weird error and I have no clue where this is coming from.

When I try to do the upload of mytest files from the Icehouse documentation (I guess it also creates a container as well), I see this error on my object node:

[root@object1 tmp]# swift upload myfiles test.txt

Object HEAD failed: http://10.7.5.11:8080/v1/AUTH_ca2e3f4... 503 Internal Server Error

And on the controller I see this error in the logs:

Oct 29 16:21:44 controller proxy-server: ERROR with Object server 10.7.5.51:6003/sda4 re: Trying to HEAD /v1/AUTH_ca2e3f4d6f8b4afa9e712aa318e1ac4c/myfiles/test.txt: Connection refused (txn: txe2222291b0864f2c9eea9-0054514c58)

Oct 29 16:21:44 controller proxy-server: Object HEAD returning 503 for [] (txn: txe2222291b0864f2c9eea9-0054514c58) (client_ip: 10.7.5.51)

So what I see that seems to be weird is that the in the controller error, ERROR with Object server 10.7.5.51:6003/sda4 it looks like its trying to get back to the object node on port 6003. Now I have no idea where that is coming from - below is my out of swift-ring-builder (there is no replication ip network and i didnt add that part of the command when creating so no idea how that is showing up either but dont think that is a problem):

[root@controller swift]# swift-ring-builder account.builder

account.builder, build version 1

262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 0.00 balance

The minimum number of hours before a partition can be reassigned is 1

Devices: id region zone ip address port replication ip replication port name weight partitions balance meta

         0       1     1       10.7.5.51  6002       10.7.5.51              6002      sda4 100.00     786432    0.00

[root@controller swift]# swift-ring-builder container.builder container.builder, build version 1 262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 0.00 balance The minimum number of hours before a partition can be reassigned is 1 Devices: id region zone ip address port replication ip replication port name weight partitions balance meta 0 1 1 10.7.5.51 6001 10.7.5.51 6002 sda4 100.00 786432 0.00

[root@controller swift]# swift-ring-builder object.builder object.builder, build version 1 262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 0.00 balance The minimum number of hours before a partition can be reassigned is 1 Devices: id region zone ip address port replication ip replication port name weight partitions balance meta 0 1 1 10.7.5.51 6000 10.7.5.51 6002 sda4 100.00 786432 0.00

So the ports are 6000, 6001, and 6002 which match with the config files on the object node. Where is port 6003 coming from? I can also do a swift stat no prob:

[root@controller swift]# swift stat

   Account: AUTH_ca2e3f4d6f8b4afa9e712aa318e1ac4c

Containers: 1

   Objects: 0

     Bytes: 0

    Accept-Ranges: bytes

    X-Timestamp: 1414516348.41483

X-Trans-Id: tx424add6ef763489f857e9-0054514fa8

  Content-Type: text ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
1

answered 2014-11-03 11:12:34 -0500

AmitA gravatar image

This maybe closed - Clay Gerrard helped me find out what wrong in my configuration. In my object.gz file I had an incorrect port in there that was causing it to fail. Changed it and now all is working!

From the relevant part of his answer:

But the thing with the port is... it has to be coming from a ring, the ring on the proxy actually, specifically /etc/swift/object.ring.gz

Get real paranoid about that.

It has to be true.

Extract object.ring.gz with gunzip and run strings on it, or use ring-builder object.ring.gz write_builder to dump what's in the ring data and figure out how the 6003 got in there. If you still don't see 6003 in the ring data then the proxy is getting rings from somewhere else because rings are the only place the proxy looks for backend ports. Check the swift_dir setting in your proxy conf. Delete the ring and restart the proxy - it shouldn't start without it.

Also since swift stat is working you can run swift stat -v to get your x-auth-token and try making some requests directly with curl. Making some object requests directly can cut out some of the noise you get when swift client makes multiple requests in response to a single cli command.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

Stats

Asked: 2014-10-30 11:36:35 -0500

Seen: 1,155 times

Last updated: Nov 03 '14