Ask Your Question
1

Insufficient Storage on swift storage nodes

asked 2014-05-21 08:53:59 -0500

shade34321 gravatar image

I have a three node set up, one node is the controller and has swift proxy installed and it also acts as another swift storage node. My two other systems are just storage nodes. I am able to run swift stat to get information and use swift post -m to add meta data as seen below.

root@xp:/etc/swift# swift stat
       Account: AUTH_dded688dc7894082b59c0b12a86d21ff
    Containers: 0
       Objects: 0
         Bytes: 0
    Meta Color: blue
   X-Timestamp: 1400619750.24388
    X-Trans-Id: txcc71cd5b54a24996bc42b-00537cab48
  Content-Type: text/plain; charset=utf-8
 Accept-Ranges: bytes

When I run

swift post python

I am getting several Insufficient Storage errors in my logs and the command is returning with a 503 error.

root@xp:/etc/swift# swift post python
Container POST failed: http://xp:8080:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python 503 Internal Server Error  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

Here is what syslog is saying

May 21 13:27:55 xp container-server: 172.16.103.90 - - [21/May/2014:13:27:55 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "tx5ff7d0f1b71d40e7a6151-00537ca9db" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3475" 0.0003
May 21 13:27:55 xp proxy-server: ERROR Insufficient Storage 172.16.103.91:6001/data (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:55 xp proxy-server: ERROR Insufficient Storage 172.16.103.92:6001/data (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:55 xp proxy-server: ERROR Insufficient Storage 172.16.103.90:6001/data (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:55 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:56 xp container-server: 172.16.103.90 - - [21/May/2014:13:27:56 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "txc5db40495c3c4cde9ef76-00537ca9dc" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3476" 0.0004
May 21 13:27:56 xp proxy-server: ERROR Insufficient Storage 172.16.103.90:6001/data (txn: txc5db40495c3c4cde9ef76-00537ca9dc)
May 21 13:27:56 xp proxy-server: ERROR Insufficient Storage 172.16.103.92:6001/data (txn: txc5db40495c3c4cde9ef76-00537ca9dc) (client_ip: 172.16.103.90)
May 21 13:27:56 xp proxy-server: ERROR Insufficient Storage 172.16.103.91:6001/data (txn: txc5db40495c3c4cde9ef76-00537ca9dc) (client_ip: 172.16.103.90)
May 21 13:27:56 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: txc5db40495c3c4cde9ef76-00537ca9dc) (client_ip: 172.16.103.90)
May 21 13:27:58 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx7b8cdae99a494f5f842b4-00537ca9de) (client_ip: 172.16.103.90)
May 21 13:28:02 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx1d8eaef7be8d4cc89600e-00537ca9e2) (client_ip: 172.16.103.90)
May 21 13:28:10 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx7da1e69e191a452c91c8c-00537ca9ea) (client_ip: 172.16.103.90)

I am getting these messages in the two other storage nodes

May 21 13:35:35 vista container-server: 172.16.103.90 - - [21/May/2014:13:35:35 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "txef7814c787b2437f97fa8-00537caba7" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3475" 0.0002
May 21 13:35:36 vista ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
2

answered 2014-08-08 16:58:11 -0500

giraffedata gravatar image

More detail: the configuration files for all the backend servers (account, container, object) contain a "devices" setting. It names a directory that contains the files that store all the data (and much of the metadata). The default, which was being used in this configuration, is /srv/node. But the user didn't actually have the files there; he had them in /data. When the files that are supposed to be there aren't, the backend server says 507 Insufficient Storage and the proxy server relays that as 503 Service Unavailable.

Thus, the fix is either to put the data at /srv/node (in this case, by mounting the disk drive that contains the data over /srv/node instead of /data) or change all the devices= to /data .

edit flag offensive delete link more

Comments

When the files that are supposed to be there aren't, what files are these?

evarlast gravatar imageevarlast ( 2016-10-04 14:36:10 -0500 )edit
0

answered 2014-05-21 15:39:34 -0500

shade34321 gravatar image

Got it figured out. I switched the mount point to /srv/node. Also I didn't see the devices option for container-server.conf or object-server.conf so adding those should have fixed it at least.

edit flag offensive delete link more

Comments

could you pls share what you have added for device mount point in the config file account-server.conf , object-server.conf and container-conf file.

vaibhavhyd2014 gravatar imagevaibhavhyd2014 ( 2015-01-05 04:02:00 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-05-21 08:53:59 -0500

Seen: 12,050 times

Last updated: May 21 '14