Ask Your Question
0

Account HEAD failed 503 Internal Server Error

asked 2013-12-04 14:53:25 -0600

vigneshb4u gravatar image

Hello all,

I am configuring Swift per the instructions in https://access.redhat.com/site/documentation/en-US/Red_Hat_OpenStack/3/html-single/Installation_and_Configuration_Guide/index.html#chap-Installing_the_OpenStack_Object_Storage_Service (https://access.redhat.com/site/docume...)

After completing configuration, I am getting a 503 internal server error while validating installation using "swift stat"

Message from syslogd@localhost at Dec 4 09:29:34 ... ?<131>proxy-server Account HEAD returning 503 for [507] (txn: tx62353887227d445fad636dbdda582119) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:35 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx88716d69ada741d1aecdec79cd7dc28e)

Message from syslogd@localhost at Dec 4 09:29:35 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx88716d69ada741d1aecdec79cd7dc28e) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:37 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: txbe9692dc2d6042b2b07de156c65db6ff)

Message from syslogd@localhost at Dec 4 09:29:37 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: txbe9692dc2d6042b2b07de156c65db6ff) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:41 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx652a3f07adb54517a6070ff35434dedc)

Message from syslogd@localhost at Dec 4 09:29:41 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx652a3f07adb54517a6070ff35434dedc) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:29:49 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx16b1fcbc6a18472bae28840b15bfacac)

Message from syslogd@localhost at Dec 4 09:29:49 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx16b1fcbc6a18472bae28840b15bfacac) (client_ip: 127.0.0.1)

Message from syslogd@localhost at Dec 4 09:30:05 ... ?<135>proxy-server Node error limited 127.0.0.1:6002 (sdb1) (txn: tx39d798d702334ebb80bd4a1d1683af07)

Message from syslogd@localhost at Dec 4 09:30:05 ... ?<131>proxy-server Account HEAD returning 503 for [] (txn: tx39d798d702334ebb80bd4a1d1683af07) (client_ip: 127.0.0.1) Account HEAD failed: http://localhost:8080/v1/AUTH_2dd248ff7c05446b801eba44c082c133 (http://localhost:8080/v1/AUTH_2dd248f...) 503 Internal Server Error

I checked all the similar questions in this forum, but nothing works.

I checked the port in both account-server.conf and also the rings. Everything looks good. I tried tempauth and got the same error.

Following are my ports in conf and ring: [DEFAULT] devices = /srv/node/1 bind_ip = 127.0.0.1 bind_port = 6002 workers = 2

swift-ring-builder /etc/swift/account.builder search /sdb1

Devices: id region zone ip address port name weight partitions balance meta 0 1 1 127.0.0.1 6002 sdb1 1.00 786432 0.00

I am not able to turn on DEBUG log for the account-server as well. This is the only log I got from the account-server log (is it a 507 error?):

Dec 4 09:29:34 localhost account-server 127.0.0.1 - - [04/Dec/2013:14:29:34 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx62353887227d445fad636dbdda582119" "-" "-" 0.0003 ""

Following is my log setting for account-server.conf

[app:account-server] use = egg:swift#account set log_level = DEBUG set log_name = account-server set log_facility = LOG_LOCAL0

Probably debug logs will help me troubleshoot ... (more)

edit retag flag offensive close merge delete

16 answers

Sort by » oldest newest most voted
0

answered 2013-12-04 22:52:45 -0600

I am not sure how closely you have followed the SAIO procedures, but the .conf files from the SAIO document are supposed to have mount_check=false in them. So you might double check the SAIO procedure you are following to be sure you have copied the contents of the various .conf files into the actual ones on your system.

Outside of SAIO documentation, one actually sets up Swift by using real disks for the storage, and so you use mount points under the path specified by the "devices" configuration file parameter.

edit flag offensive delete link more
0

answered 2013-12-04 22:32:25 -0600

vigneshb4u gravatar image

Gosh! This problem got resolved after I set "mount_check = false" for account, container and object servers.

swift stat

Account: AUTH_2dd248ff7c05446b801eba44c082c133 Containers: 0 Objects: 0 Bytes: 0 Accept-Ranges: bytes X-Timestamp: 1386196007.43202 Content-Type: text/plain; charset=utf-8

Nowhere in the configuration guide, "mount_check = false" settings was mentioned. Was stuck with this problem for 4 days.

Thanks a lot Peter, for all your time and expertise.

I didn't understand what you meant by "since you don't have individual device mount points". Do you mind elborating?

edit flag offensive delete link more
0

answered 2013-12-04 22:22:21 -0600

vigneshb4u gravatar image

Sorry I didn't get what you meant in the first line. Could you please elaborate?

I suppose by device "names", you mean "sdb1" in my case, and I have already created under those directories. for S in {1..4}; do mkdir -p /srv/node/${S}/sdb${S}; done

Yes I am following SAIO procedure for a sandbox setup.

I built the rings as follows: swift-ring-builder /etc/swift/object.builder create 18 3 1 swift-ring-builder /etc/swift/container.builder create 18 3 1 swift-ring-builder /etc/swift/account.builder create 18 3 1 swift-ring-builder /etc/swift/account.builder add z1-127.0.0.1:6002/sdb1 1 swift-ring-builder /etc/swift/container.builder add z1-127.0.0.1:6001/sdb1 1 swift-ring-builder /etc/swift/object.builder add z1-127.0.0.1:6000/sdb1 1 swift-ring-builder /etc/swift/account.builder rebalance swift-ring-builder /etc/swift/container.builder rebalance swift-ring-builder /etc/swift/object.builder rebalance

edit flag offensive delete link more
0

answered 2013-12-04 22:15:35 -0600

Yes, you need to now have device "names" that you use with your ring files that are directories under /srv/node/*/.

And since you don't have individual device mount points, you should probably set "mount_check = false" for account, container and object servers as well.

Are you following the SAIO procedure some how?

How are you building the rings?

edit flag offensive delete link more
0

answered 2013-12-04 21:53:27 -0600

vigneshb4u gravatar image

Following is the content of /etc/fstab

/dev/mapper/vg_openstack-lv_root / ext4 defaults 1 1 UUID=d5049cfa-b43d-489f-95fa-ba5ebb5d7d8e /boot ext4 defaults 1 2 /dev/mapper/vg_openstack-lv_home /home ext4 acl,user_xattr 1 2 /dev/mapper/vg_openstack-lv_swap swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0

Following is how I configured the device_mountpoint

mkdir /home/storage chown swift:swift /home/storage/ mkdir /home/storage/{1..4} chown swift:swift /home/storage/* mkdir /srv/node/ ln -s /home/storage/{1..4} /srv/node for S in {1..4}; do mkdir -p /srv/node/${S}/sdb${S}; done chown -R swift:swift /etc/swift/ /srv/node/ chown -R swift:swift /etc/swift/ /srv/node/{1..4} chown -R swift:swift /etc/swift/ /srv/node/{1..4}/* restorecon -R /srv

As I mentioned earlier I set "devices = /srv/node/1" in the account-server.conf, object-server.conf and container-server.conf.

Do you see any issues with the above procedure?

edit flag offensive delete link more
0

answered 2013-12-04 21:43:29 -0600

vigneshb4u gravatar image

Sure, Here you go..

df

Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_openstack-lv_root 51606140 9571576 39413124 20% / tmpfs 49568868 232 49568636 1% /dev/shm /dev/sda1 495844 38044 432200 9% /boot /dev/mapper/vg_openstack-lv_home 1097480696 234232 1041497652 1% /home /dev/sr0 3632776 3632776 0 100% /media/RHEL_6.4 x86_64 Disc 1

edit flag offensive delete link more
0

answered 2013-12-04 21:40:00 -0600

Dec 4 15:57:30 localhost account-server 127.0.0.1 - - [04/Dec/2013:20:57:30 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx2929b3818f334ece8047d43b80bb727c" "-" "-" 0.0003 ""

So this means there is a problem accessing your devices. Can you provide the output of df?

edit flag offensive delete link more
0

answered 2013-12-04 21:34:49 -0600

vigneshb4u gravatar image

I will create a bug as you suggest but am not sure if this is a bug.

I just followed the installation and configuration guide of the Openstack grizzly and I am not able to have swift up and running. So I still think this is a problem with my configuration than a bug.

BTW, following is the complete log in the proxy-server.log when I ran "swift stat"

Dec 4 15:57:30 localhost proxy-server Authenticating user token Dec 4 15:57:30 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:30 localhost proxy-server Storing 0756e11b08a0053c5549aa81274d7347 token in memcache Dec 4 15:57:30 localhost account-server 127.0.0.1 - - [04/Dec/2013:20:57:30 +0000] "HEAD /sdb1/23746/AUTH_2dd248ff7c05446b801eba44c082c133" 507 - "tx2929b3818f334ece8047d43b80bb727c" "-" "-" 0.0003 "" Dec 4 15:57:31 localhost proxy-server Authenticating user token Dec 4 15:57:31 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:31 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d7347 Dec 4 15:57:33 localhost proxy-server Authenticating user token Dec 4 15:57:33 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:33 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d73 47 Dec 4 15:57:37 localhost proxy-server Authenticating user token Dec 4 15:57:37 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:37 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d7347 Dec 4 15:57:45 localhost proxy-server Authenticating user token Dec 4 15:57:45 localhost proxy-server Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role Dec 4 15:57:45 localhost proxy-server Returning cached token 0756e11b08a0053c5549aa81274d7347

edit flag offensive delete link more
0

answered 2013-12-04 21:25:51 -0600

Okay, so you are using Grizzly, not Folsom.

Can you file a Bugzilla here: https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20OpenStack (https://bugzilla.redhat.com/enter_bug...)

And then attach to that BZ the contents of your /var/log/proxyserver.log file?

This does not sound like a swift community, or upstream, issue, but rather an issue with the Red Hat OpenStack product.

edit flag offensive delete link more
0

answered 2013-12-04 21:02:26 -0600

vigneshb4u gravatar image

I don't have a clue why Traceback logs are working. Those logs would definitely help me pin down the root cause.

Following are the versions of the rpm.. Most are 1.8.0. Probably I should install newer version of python-swiftclient?

rpm -qa | grep "swift"

openstack-swift-proxy-1.8.0-2.el6.noarch python-swiftclient-1.4.0-1.el6.noarch openstack-swift-1.8.0-2.el6.noarch openstack-swift-account-1.8.0-2.el6.noarch openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch openstack-swift-container-1.8.0-2.el6.noarch openstack-swift-object-1.8.0-2.el6.noarch

In case of upgrading to newer version, will I need to redo the entire swift configuration, like building rings etc?

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-12-04 14:53:25 -0600

Seen: 1,047 times

Last updated: Dec 04 '13