Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Insufficient Storage on swift storage nodes

I have a three node set up, one node is the controller and has swift proxy installed and it also acts as another swift storage node. My two other systems are just storage nodes. I am able to run swift stat to get information and use swift post -m to add meta data as seen below.

root@xp:/etc/swift# swift stat
       Account: AUTH_dded688dc7894082b59c0b12a86d21ff
    Containers: 0
       Objects: 0
         Bytes: 0
    Meta Color: blue
   X-Timestamp: 1400619750.24388
    X-Trans-Id: txcc71cd5b54a24996bc42b-00537cab48
  Content-Type: text/plain; charset=utf-8
 Accept-Ranges: bytes

When I run

swift post python

I am getting several Insufficient Storage errors in my logs and the command is returning with a 503 error.

root@xp:/etc/swift# swift post python
Container POST failed: http://xp:8080:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python 503 Internal Server Error  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

Here is what syslog is saying

May 21 13:27:55 xp container-server: 172.16.103.90 - - [21/May/2014:13:27:55 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "tx5ff7d0f1b71d40e7a6151-00537ca9db" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3475" 0.0003
May 21 13:27:55 xp proxy-server: ERROR Insufficient Storage 172.16.103.91:6001/data (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:55 xp proxy-server: ERROR Insufficient Storage 172.16.103.92:6001/data (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:55 xp proxy-server: ERROR Insufficient Storage 172.16.103.90:6001/data (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:55 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx5ff7d0f1b71d40e7a6151-00537ca9db)
May 21 13:27:56 xp container-server: 172.16.103.90 - - [21/May/2014:13:27:56 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "txc5db40495c3c4cde9ef76-00537ca9dc" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3476" 0.0004
May 21 13:27:56 xp proxy-server: ERROR Insufficient Storage 172.16.103.90:6001/data (txn: txc5db40495c3c4cde9ef76-00537ca9dc)
May 21 13:27:56 xp proxy-server: ERROR Insufficient Storage 172.16.103.92:6001/data (txn: txc5db40495c3c4cde9ef76-00537ca9dc) (client_ip: 172.16.103.90)
May 21 13:27:56 xp proxy-server: ERROR Insufficient Storage 172.16.103.91:6001/data (txn: txc5db40495c3c4cde9ef76-00537ca9dc) (client_ip: 172.16.103.90)
May 21 13:27:56 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: txc5db40495c3c4cde9ef76-00537ca9dc) (client_ip: 172.16.103.90)
May 21 13:27:58 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx7b8cdae99a494f5f842b4-00537ca9de) (client_ip: 172.16.103.90)
May 21 13:28:02 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx1d8eaef7be8d4cc89600e-00537ca9e2) (client_ip: 172.16.103.90)
May 21 13:28:10 xp proxy-server: Container POST returning 503 for (503, 503, 503) (txn: tx7da1e69e191a452c91c8c-00537ca9ea) (client_ip: 172.16.103.90)

I am getting these messages in the two other storage nodes

May 21 13:35:35 vista container-server: 172.16.103.90 - - [21/May/2014:13:35:35 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "txef7814c787b2437f97fa8-00537caba7" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3475" 0.0002
May 21 13:35:36 vista container-server: 172.16.103.90 - - [21/May/2014:13:35:36 +0000] "POST /data/14/AUTH_dded688dc7894082b59c0b12a86d21ff/python" 507 - "tx9453ac8eb5cb40f9b6a4f-00537caba8" "POST http://xp:8080/v1/AUTH_dded688dc7894082b59c0b12a86d21ff/python" "proxy-server 3476" 0.0002
May 21 13:35:52 vista object-auditor: Begin object audit "forever" mode (ZBF)
May 21 13:35:52 vista object-auditor: Object audit (ZBF) "forever" mode completed: 0.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 0.00, Total bytes/sec: 0.00, Auditing time: 0.00, Rate: 0.00
May 21 13:35:52 vista object-auditor: Begin object audit "forever" mode (ALL)
May 21 13:35:52 vista object-auditor: Object audit (ALL) "forever" mode completed: 0.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 0.00, Total bytes/sec: 0.00, Auditing time: 0.00, Rate: 0.00
May 21 13:35:52 vista object-auditor: Begin object audit "forever" mode (ZBF)
May 21 13:35:52 vista object-auditor: Object audit (ZBF) "forever" mode completed: 0.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 0.00, Total bytes/sec: 0.00, Auditing time: 0.00, Rate: 0.00
May 21 13:35:53 vista object-replicator: Starting object replication pass.
May 21 13:35:53 vista object-replicator: data is not mounted

My mount points for all of my extra storage space is the same across all three machines, /data. Each has it's own hard drive specifically for this

In my account-server.conf I have devices = / to account for the different mount point. I also have the bind port as 6002. For container-server.conf I have bind port as 6001 and bind_ip as 0.0.0.0. For object-server.conf I have the same thing but bind port is 6000. These port numbers were taken from the installation guide during the ring creating process.

Here is the output of df and ls -lah on the controller:

root@xp:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       451G  2.2G  426G   1% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           396M  700K  395M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G  4.0K  2.0G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       466G   33M  466G   1% /data
root@xp:/# ls -lah /data/
total 4.0K
drwxrwxrwx  4 swift swift   31 May 20 21:02 .
drwxr-xr-x 24 root  root  4.0K May 19 21:34 ..
drwxr-xr-x  3 swift swift   16 May 20 21:02 accounts
drwxr-xr-x  2 swift swift    6 May 20 21:02 tmp

And here it is for one of the other storage nodes.

root@seven:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       451G  2.0G  427G   1% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           787M  684K  787M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G  4.0K  3.9G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1        60M  3.4M   56M   6% /boot/efi
/dev/sdb1       932G   33M  932G   1% /data
root@seven:~# ls -lah /data
total 4.0K
drwxrwxrwx  4 swift swift   31 May 20 21:02 .
drwxr-xr-x 24 root  root  4.0K May 19 21:35 ..
drwxr-xr-x  3 swift swift   16 May 20 21:02 accounts
drwxr-xr-x  2 swift swift    6 May 20 21:02 tmp

Any suggestions on what I messed up? Also sorry for all the pasting but it doesn't like me putting in links. Thanks for the help!