Ask Your Question
0

Account HEAD failed 503 Internal Server Error

asked 2013-12-04 14:53:25 -0600

vigneshb4u gravatar image

Hello all,

I am configuring Swift per the instructions in https://access.redhat.com/site/documentation/en-US/Red_Hat_OpenStack/3/html-single/Installation_and_Configuration_Guide/index.html#chap-Installing_the_OpenStack_Object_Storage_Service (https://access.redhat.com/site/docume...)

After completing configuration, I am getting a 503 internal server error while validating installation using "swift stat"

Message from syslogd@localhost at Dec 4 09:29:34 ... ?<131>proxy-server Account HEAD returning 503 for [507] (txn: tx62353887227d445fad636dbdda582119) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:35 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx88716d69ada741d1aecdec79cd7dc28e)

Message from syslogd@localhost at Dec 4 09:29:35 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx88716d69ada741d1aecdec79cd7dc28e) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:37 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: txbe9692dc2d6042b2b07de156c65db6ff)

Message from syslogd@localhost at Dec 4 09:29:37 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: txbe9692dc2d6042b2b07de156c65db6ff) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:41 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx652a3f07adb54517a6070ff35434dedc)

Message from syslogd@localhost at Dec 4 09:29:41 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx652a3f07adb54517a6070ff35434dedc) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:49 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx16b1fcbc6a18472bae28840b15bfacac)

Message from syslogd@localhost at Dec 4 09:29:49 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx16b1fcbc6a18472bae28840b15bfacac) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:30:05 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx39d798d702334ebb80bd4a1d1683af07)

Message from syslogd@localhost at Dec 4 09:30:05 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx39d798d702334ebb80bd4a1d1683af07) (client_ip: 127.0.0.1) Account HEAD failed: http://localhost:8080/v1/AUTH_2dd248ff7c05446b801eba44c082c133 (http://localhost:8080/v1/AUTH_2dd248f...) 503 Internal Server Error

I checked all the similar questions in this forum, but nothing works.

I checked the port in both account-server.conf and also the rings. Everything looks good. I tried tempauth and got the same error.

Following are my ports in conf and ring: [DEFAULT] devices = /srv/node/1 bind_ip = 127.0.0.1 bind_port = 6002 workers = 2

swift-ring-builder /etc/swift/account.builder search /sdb1

Devices: id region zone ip address port name weight partitions balance meta 0 1 1 127.0.0.1 6002 sdb1 1.00 786432 0.00

I am not able to turn on DEBUG log for the account-server as well. This is the only log I got from the account-server log (is it a 507 error?):

Dec 4 09:29:34 localhost account-server 127.0.0.1 - - [04/Dec/2013:14:29:34 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx62353887227d445fad636dbdda582119" "-" "-" 0.0003 ""

Following is my log setting for account-server.conf

[app:account-server] use = egg:swift#account set log_level = DEBUG set log_name = account-server set log_facility = LOG_LOCAL0

Probably debug logs will help me troubleshoot ... (more)

edit retag flag offensive close merge delete

16 answers

Sort by ยป oldest newest most voted
0

answered 2013-12-04 16:25:40 -0600

vigneshb4u gravatar image

No, I do not see any "Traceback" messages. I think you mean some python exceptions. I don't see them.

edit flag offensive delete link more
0

answered 2013-12-04 16:29:07 -0600

Can you also provide your account, container and object server conf files?

How do you have syslog configured?

And what version of Swift are you using?

edit flag offensive delete link more
0

answered 2013-12-04 16:43:02 -0600

vigneshb4u gravatar image

/etc/swift/account-server.conf

[DEFAULT] devices = /srv/node/1 bind_ip = 127.0.0.1 bind_port = 6002 workers = 2

[pipeline:main] pipeline = account-server

[app:account-server] use = egg:swift#account set log_level = DEBUG set log_name = account-server set log_facility = LOG_LOCAL0

[account-replicator]

[account-auditor]

[account-reaper]

/etc/swift/object-server.conf

[DEFAULT] devices = /srv/node/1 bind_ip = 127.0.0.1 bind_port = 6000 workers = 3

[pipeline:main] pipeline = object-server

[app:object-server] use = egg:swift#object

[object-replicator]

[object-updater]

[object-auditor]

/etc/swift/container-server.conf

[DEFAULT] devices = /srv/node/1 bind_ip = 127.0.0.1 bind_port = 6001 workers = 2

[pipeline:main] pipeline = container-server

[app:container-server] use = egg:swift#container

[container-replicator]

[container-updater]

[container-auditor]

[container-sync]

Following are the RULES in /etc/rsyslog.conf

RULES

Log all kernel messages to the console.

Logging much else clutters up the screen.

#kern.* /dev/console

Log anything (except mail) of level info or higher.

Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none /var/log/messages

The authpriv file has restricted access.

authpriv.* /var/log/secure

Log all the mail messages in one place.

mail.* -/var/log/maillog

Log cron stuff

cron.* /var/log/cron

Everybody gets emergency messages

*.emerg *

Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

Save boot messages also to boot.log

local7.* /var/log/boot.log

local0.* /var/log/proxyserver.log

swift --version

swift 1.4.0

Please let me know if you need more information.

edit flag offensive delete link more
0

answered 2013-12-04 15:27:55 -0600

vigneshb4u gravatar image

Following is my proxy-server.conf

[DEFAULT] bind_port = 8080 workers = 8 user = swift

[pipeline:main] pipeline = healthcheck cache authtoken keystone proxy-server

[app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true set log_level = DEBUG set log_name = proxy-server set log_facility = LOG_LOCAL0

[filter:cache] use = egg:swift#memcache memcache_servers = 127.0.0.1:11211

[filter:catch_errors] use = egg:swift#catch_errors

[filter:healthcheck] use = egg:swift#healthcheck

[filter:keystone] use = egg:swift#keystoneauth operator_roles = Member,admin,SwiftOperator is_admin = true cache = swift.cache

[filter:authtoken] paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory admin_tenant_name = services admin_token = 49150054e2ec4636c511 auth_token = 49150054e2ec4636c511 admin_user = swift admin_password = * auth_host = localhost auth_port = 35357 auth_protocol = http signing_dir = /tmp/keystone-signing-swift

edit flag offensive delete link more
0

answered 2013-12-04 16:19:19 -0600

Do you see any "Traceback" messages in your log files?

edit flag offensive delete link more
0

answered 2013-12-04 20:43:19 -0600

From the rsyslog.conf file it looks like all of the account server and proxy server messages should be landing in /var/log/proxyserver.log, and yet you don't see any Tracebacks in there. So something is off as to why you can't see them.

You are using 1.4.0, that seems kind of old, is there a way you can use something newer like, 1.10.0, Havana?

edit flag offensive delete link more
0

answered 2013-12-04 21:02:26 -0600

vigneshb4u gravatar image

I don't have a clue why Traceback logs are working. Those logs would definitely help me pin down the root cause.

Following are the versions of the rpm.. Most are 1.8.0. Probably I should install newer version of python-swiftclient?

rpm -qa | grep "swift"

openstack-swift-proxy-1.8.0-2.el6.noarch python-swiftclient-1.4.0-1.el6.noarch openstack-swift-1.8.0-2.el6.noarch openstack-swift-account-1.8.0-2.el6.noarch openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch openstack-swift-container-1.8.0-2.el6.noarch openstack-swift-object-1.8.0-2.el6.noarch

In case of upgrading to newer version, will I need to redo the entire swift configuration, like building rings etc?

edit flag offensive delete link more
0

answered 2013-12-04 21:25:51 -0600

Okay, so you are using Grizzly, not Folsom.

Can you file a Bugzilla here: https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20OpenStack (https://bugzilla.redhat.com/enter_bug...)

And then attach to that BZ the contents of your /var/log/proxyserver.log file?

This does not sound like a swift community, or upstream, issue, but rather an issue with the Red Hat OpenStack product.

edit flag offensive delete link more
0

answered 2013-12-04 21:34:49 -0600

vigneshb4u gravatar image

I will create a bug as you suggest but am not sure if this is a bug.

I just followed the installation and configuration guide of the Openstack grizzly and I am not able to have swift up and running. So I still think this is a problem with my configuration than a bug.

BTW, following is the complete log in the proxy-server.log when I ran "swift stat"

Dec 4 15:57:30 localhost proxy-server Authenticating user token Dec 4 15:57:30 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:30 localhost proxy-server Storing 0756e11b08a0053c5549aa81274d7347 token in memcache Dec 4 15:57:30 localhost account-server 127.0.0.1 - - [04/Dec/2013:20:57:30 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx2929b3818f334ece8047d43b80bb727c" "-" "-" 0.0003 "" Dec 4 15:57:31 localhost proxy-server Authenticating user token Dec 4 15:57:31 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:31 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d7347 Dec 4 15:57:33 localhost proxy-server Authenticating user token Dec 4 15:57:33 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:33 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d73 47 Dec 4 15:57:37 localhost proxy-server Authenticating user token Dec 4 15:57:37 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:37 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d7347 Dec 4 15:57:45 localhost proxy-server Authenticating user token Dec 4 15:57:45 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:45 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d7347

edit flag offensive delete link more
0

answered 2013-12-04 21:40:00 -0600

Dec 4 15:57:30 localhost account-server 127.0.0.1 - - [04/Dec/2013:20:57:30 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx2929b3818f334ece8047d43b80bb727c" "-" "-" 0.0003 ""

So this means there is a problem accessing your devices. Can you provide the output of df?

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-12-04 14:53:25 -0600

Seen: 1,028 times

Last updated: Dec 04 '13