Elasticsearch day-to-day operations and failure scenarios (8.x / 9.x)

Not an installation: a scenario-based runbook for running an installed Elasticsearch 8.x / 9.x cluster day to day and recovering it when it breaks. Health and allocation explain, resolving unassigned shards (disk, filtering, retry_failed, replicas, stale/empty primary as a last resort), disk watermarks, the read_only_allow_delete block, adding and safely removing nodes, zero-downtime reindex with an alias, ILM and rollover, _shrink / _split, restoring a single index from a snapshot under a new name. If you do not know what to do, start with the health check. Snapshot repository setup and SLM, security hardening and version upgrades are in separate Elasticsearch guides.

Topology: Elasticsearch 8.x / 9.x cluster (1–N nodes), TLS and authentication enabled

Scenarios

• Cluster health: status, nodes, unassigned shards, disk — Read-only: green/yellow/red, node list, unassigned (UNASSIGNED) shards and their cause (allocation explain), disk usage and watermarks, pending tasks. Start here if you are not sure what to do.
• Resolve an unassigned (UNASSIGNED) shard (based on explain) — The cluster is yellow or red and shards are not being allocated: depending on the allocation explain cause, disk, allocation filtering, the retry limit (retry_failed), the replica count or, as a last resort, a primary from an old copy.
• View / change the disk watermark settings — Shards are being moved or not allocated because of disk usage; the watermark thresholds (low / high / flood_stage) are to be changed temporarily or permanently, or reset to the defaults.
• Index became read-only: remove the read_only_allow_delete block — Writes are rejected with 'FORBIDDEN/12/index read-only / allow delete (api)' or 'TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark': free up disk first, then remove the block.
• Add a node to the cluster (enrollment token) — More capacity is needed: Elasticsearch has been installed on a new server, which joins the cluster with an enrollment token; shards rebalance automatically. No downtime.
• Remove a node safely (exclude, drain, shut down) — A node is being retired or its hardware replaced: its shards are moved to the other nodes with an allocation exclude, the node is verified to be empty, it is given a voting exclusion if it is master-eligible, and then it is shut down.
• Reindex: zero-downtime cutover to a new index with an alias for a mapping/settings change — A mapping field type, the analyzer or the shard count must change: a new index is created, the data is copied with _reindex, and the alias is switched atomically to the new index.
• Set up / troubleshoot index lifecycle (ILM) and rollover — Log/time series indices keep growing or are not deleted: an ILM policy (rollover + delete), an index template and the initial index (write alias) are set up; explain shows why an existing policy is not progressing.
• Fix the shard count: _shrink (reduce) / _split (increase) — The index's shard count was chosen badly: very small shards (oversharding) are merged with _shrink, very large shards are divided with _split; the cutover is done with an alias.
• Restore a single index from a snapshot under a new name — An index was deleted or corrupted: only that index is restored from the snapshot repository under a new name (rename) without touching the live index; after verification it is put into service with an alias.

Before you start

• On … (…) you have root access, curl and the cluster's HTTP CA certificate (…); you have the password of the elastic user (or of a user with manage + index privileges). The API calls are made to this node over HTTPS.
• A change record has been opened and the application team informed. Shard relocation, reindex and resize operations put load on disk, network and CPU; do them outside peak hours. Before irreversible steps (deletion, stale/empty primary) there must be a current snapshot: Snapshot.
• The new server is prepared with the operating system preparation and software installation phases of the cluster installation guide: the same Elasticsearch version, vm.max_map_count, heap, swap disabled, 9200/9300 open in the firewall; the service has never been started (enrollment is only possible before the first start).

Phases of the guide

• API access — Temporary credentials file for the API
• Status: health, nodes, shards, disk (read-only) — Cluster health, nodes and unassigned shards · Ask why the shard is unassigned (allocation explain) · Disk usage and watermarks · Pending tasks and interpreting the result
• 1. Diagnosis: why is the shard not allocated? — The index's shards and the explain decision
• 2. Apply — Retry failed allocations (retry_failed) · Match the replica count to the node count · Remove allocation filtering · Go to the disk block · Allocate a primary from an old copy (allocate_stale_primary) · Allocate an empty primary (allocate_empty_primary)
• 3. Verification — Watch the shards start
• 1. Diagnosis — Disk usage and current thresholds
• 2. Apply — …Change the thresholds…Reset the thresholds to the defaults…
• 1. Diagnosis: block and disk — View the blocked indices and the disk · Options for freeing disk space
• 2. Apply — Remove the block
• 3. Verification — Test that writes work
• 1. Diagnosis — View the cluster status
• 2. Apply: enrollment and joining — Create a node enrollment token · Configure the new node with the token · Check node.name and the network settings · Start the service
• 3. Verification — Verify that the node has joined and receives shards
• 1. Diagnosis — View the node's shards and role
• 2. Apply: drain and shut down — Move the shards off the node (allocation exclude) · Wait for the node to drain · Remove the master-eligible node from voting · Stop the node · Remove the exclude and the voting exclusion
• 3. Verification — Verify cluster health
• 1. Diagnosis and preparation — View the source index and the alias · Create the new index with the new mapping/settings
• 2. Apply: copy and switch the alias — Copy the data (_reindex, in the background) · Monitor the task · Prepare the new index and compare the counts · Switch the alias atomically to the new index
• 3. Afterwards — Close the old index (delete it later)
• 1. ILM policy, template and initial index — Create the ILM policy · Create the index template · Create the initial index with the write alias
• Diagnosis: why is ILM not progressing? — View the ILM status and the failing indices · Retry the failed step
• Manual rollover — Roll over the alias now
• 1. Diagnosis — The index's shard count and size
• 3. Apply — …Shrink…Split…: … → … (… shards) · Switch the alias to the new index
• 4. Afterwards — Close the source index (delete it later)
• 1. Diagnosis: is the index in the snapshot? — View the snapshots and the index inside
• 2. Apply: restore under a new name — Restore the index as …… · Verify and put into service
• Cleanup — Delete the credentials file