Ask Your Question

peter-a-portante's profile - activity

2013-12-04 22:52:45 -0600 answered a question Account HEAD failed 503 Internal Server Error

I am not sure how closely you have followed the SAIO procedures, but the .conf files from the SAIO document are supposed to have mount_check=false in them. So you might double check the SAIO procedure you are following to be sure you have copied the contents of the various .conf files into the actual ones on your system.

Outside of SAIO documentation, one actually sets up Swift by using real disks for the storage, and so you use mount points under the path specified by the "devices" configuration file parameter.

2013-12-04 22:15:35 -0600 answered a question Account HEAD failed 503 Internal Server Error

Yes, you need to now have device "names" that you use with your ring files that are directories under /srv/node/*/.

And since you don't have individual device mount points, you should probably set "mount_check = false" for account, container and object servers as well.

Are you following the SAIO procedure some how?

How are you building the rings?

2013-12-04 21:40:00 -0600 answered a question Account HEAD failed 503 Internal Server Error

Dec 4 15:57:30 localhost account-server 127.0.0.1 - - [04/Dec/2013:20:57:30 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx2929b3818f334ece8047d43b80bb727c" "-" "-" 0.0003 ""

So this means there is a problem accessing your devices. Can you provide the output of df?

2013-12-04 21:25:51 -0600 answered a question Account HEAD failed 503 Internal Server Error

Okay, so you are using Grizzly, not Folsom.

Can you file a Bugzilla here: https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20OpenStack (https://bugzilla.redhat.com/enter_bug...)

And then attach to that BZ the contents of your /var/log/proxyserver.log file?

This does not sound like a swift community, or upstream, issue, but rather an issue with the Red Hat OpenStack product.

2013-12-04 20:43:19 -0600 answered a question Account HEAD failed 503 Internal Server Error

From the rsyslog.conf file it looks like all of the account server and proxy server messages should be landing in /var/log/proxyserver.log, and yet you don't see any Tracebacks in there. So something is off as to why you can't see them.

You are using 1.4.0, that seems kind of old, is there a way you can use something newer like, 1.10.0, Havana?

2013-12-04 16:29:07 -0600 answered a question Account HEAD failed 503 Internal Server Error

Can you also provide your account, container and object server conf files?

How do you have syslog configured?

And what version of Swift are you using?

2013-12-04 16:19:19 -0600 answered a question Account HEAD failed 503 Internal Server Error

Do you see any "Traceback" messages in your log files?

2013-11-13 12:46:44 -0600 answered a question Trying to get final status of PUT

That message usually means the object server, found at 10.10.2.22:6000 for device /sdo1, did not respond within 10s of the proxy-server's request.

If you take the transaction ID, txf1bd783ca1d344dfbe5ca-00528356c8, and search all of the logs, you might find the log messages in the object server to see how long it actually took (if it was successful). If you can't find that transaction ID, you might want to check the logging level set for the object servers to ensure INFO messages are logged.

2013-10-16 15:54:28 -0600 answered a question Why does my cluster see so many errors?

Hi Tim,

For the two error logs that you posted, the first appears to be a 10MB file, of which 6,553,600 (or 100 64KB chunks) have been transferred, and then the proxy server SUCCESSFULLY declared the transfer finished because the object server did not respond for the 101st chunk. The client should see that the connection was closed prematurely with missing data, or might timeout waiting for more data if connection keep-alive was used.

The second of the two error logs is similar to the first, except that it was for a 4GB file, which had only 1 chunk transferred before the object server timed out.

Raising your node_timeout value in the proxy server configuration might help us determine the scope of the problems on the object server.

You have another error in the big log you posted, which seems to be reporting that the proxy server cannot connect to an object server. Are there other errors on the system that might correlate with the object servers? Are these logs all taken from one system, or from all your storage nodes?

I would suggest raising the number of object workers, doubling or tripling it to see if that helps, and if it doesn't, it might help us understand the behavior we are dealing with depending on what results.

2013-10-12 13:42:55 -0600 answered a question Why does my cluster see so many errors?

Please pardon all these questions, if you have already answered them (and so I just could not dig through the history to pull out the answers well enough).

Is it the same error as what you first reported? 10s ChunkReadTimeout? Can you post them?

Can I assume that each node in this swift configuration is running proxy, account, container, and object servers? I think the object server worker counts are at 8, is that still the case? Did you try to set max_clients lower, raising the number of workers? Did you try ThreadPools (and for that matter, what version of Swift are you running)?

Can you post the output of "xfs_info" on a disk? We have not seen much benefit, if at all, with inode sizes of 1,024. Additionally, for large disks, the default allocation groups that XFS uses can be kinda small (by default 4 allocation groups per 1 TB, so hoping it is at 16 or higher for your disks). And depending on how big your partitions on a disk grow to, you might benefit from a larger name size, 8192 or higher depending.

Are you using 1ge or 10ge networking? And do you a single nic for all traffic, front and backend, or do you separate your client traffic from your inter-swift-node traffic.

I don't think memcache is a factor here, but curious about where your memcache servers are located?

Finally, can you post a one minute capture of just your proxy logs, scrubbed?

2013-10-10 12:17:29 -0600 answered a question Do I need to reinstall swift SAIO when changing from loopback to a partition?

You have to at least make sure the resetswift script does the right thing. I don't believe you need to change the .conf files, unless you have deviated from the SAIO for other reasons.