DBAtlas · Database operations atlas

Manage disk watermarks and remove the flood-stage write block

ElasticsearchCluster/Node

Command

GET _cat/allocation?v&h=node,shards,disk.indices,disk.used,disk.avail,disk.percent&s=disk.percent:desc
GET _cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.disk
# Adjust thresholds temporarily (null returns to the default):
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}
# After freeing space, remove the flood-stage write block:
PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": null
}

What it does

Shows disk usage per node and the effective watermark values; adjusts the low/high/flood_stage thresholds and removes the read_only_allow_delete block placed on indices at flood stage.

When to use it

When writes are rejected with 'TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block', new shards are not allocated, or shards start moving off a full node.

Caution

Above low, no new shards are allocated to the node; above high, shards are moved off the node; above flood_stage, indices with a shard on that node become read-only. Raising thresholds only buys time; the lasting fix is deleting old indices, reducing replicas or adding disk/nodes. If the disk fills completely, shards can be corrupted.

Open in DBAtlas →

Related