Ask Your Question

What happens if a device is completely out of disk space?

asked 2011-03-22 21:09:22 -0500

rob-janney gravatar image

Will data destined for partitions on this device fail to be stored? Or will the data be stored to another partition on another device with available space?

Is there any manual procedure necessary for recovering from this condition? (aside from http delete calls to reclaim space before more objects can be stored)

Is there a way to set a threshold to stop storing at 1%/etc disk free or does the system already do this internally?

edit retag flag offensive close merge delete

4 answers

Sort by ยป oldest newest most voted

answered 2011-03-23 16:15:54 -0500

clay-gerrard gravatar image

Handoffs will ensure the data is stored. An object server out of space will respond to a PUT with a 507. As one long as the object can get written to at least ONE of the primary replica nodes - the condition should be transparent to the client and data durability won't get compromised. The handoff node will keep trying to replicate the partition off of itself tho..

Ultimately you'll need to rebalance the ring ( ( ). The specific device(s) that are over-loaded will need to have their 'weight' dropped down and the partitions will get shuffled around. Once you push out the new ring the "full" devices will start to replicate partitions that no longer belong there off to other devices.

There is currently no mechanism for setting a threshold that would automatically make changes to the ring.

edit flag offensive delete link more

answered 2011-03-24 00:18:11 -0500

rob-janney gravatar image

Thanks! That answers my question. We definitely will be re-balancing & trying to prevent this from happening.

@JaY It hasn't gotten over 80% full. I was just curious about the failure mode in the event of this happening.

edit flag offensive delete link more

answered 2011-03-24 00:18:17 -0500

rob-janney gravatar image

Thanks clayg, that solved my question.

edit flag offensive delete link more

answered 2011-03-23 18:29:34 -0500

letterj gravatar image

As a best practice, it is recommended that when your cluster gets 80% full you add additional storage. How full did your entire cluster get?

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2011-03-22 21:09:22 -0500

Seen: 161 times

Last updated: Mar 24 '11