Elasticsearch resilience: replica + snapshot + CCR

Symptom

Goal: build resilience for Elasticsearch with replica shards, dedicated master/data nodes, snapshot backups (SLM), and cross-cluster replication (CCR).

1. Separate roles: dedicated master and data nodes

For cluster stability, voting master-eligible nodes (preferably 3, an odd majority) are separated from the data workload. Assign roles with node.roles in elasticsearch.yml and define seed hosts for discovery. 3 masters prevent split-brain.

Elasticsearch

# elasticsearch.yml on master nodes:
# node.roles: [ master ]
# on data nodes:
# node.roles: [ data, ingest ]
# discovery.seed_hosts: [ m1, m2, m3 ]
# cluster.initial_master_nodes: [ m1, m2, m3 ]

2. Set the index replica count

number_of_replicas is the number of copies kept for each primary shard; it should be at least 1 so that data stays available when a data node is lost. It can be changed dynamically on a live index; the replica count must be less than the number of data nodes.

Elasticsearch

PUT /orders/_settings
{
  "index": { "number_of_replicas": 1 }
}
# default via template (composable template; the old _template API is legacy):
# PUT _index_template/order-tpl { "index_patterns": ["order*"], "template": { "settings": { "number_of_replicas": 1 } } }

3. Define a snapshot repository

Backups are taken to a repository: a shared file system (fs) or the S3/GCS/Azure plugins. To be able to recover data even if the cluster is deleted, the repository must be on durable storage outside the cluster.

Elasticsearch

PUT /_snapshot/backup_repo
{
  "type": "fs",
  "settings": { "location": "/mnt/es_backups", "compress": true }
}

4. Schedule automatic snapshots with SLM

Snapshot Lifecycle Management (SLM) takes regular snapshots and applies a retention policy. A cron expression defines the frequency, which indices are included, and how many snapshots are kept.

Elasticsearch

PUT /_slm/policy/daily_backup
{
  "schedule": "0 30 1 * * ?",
  "name": "<daily-{now/d}>",
  "repository": "backup_repo",
  "config": { "indices": ["*"] },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}

5. Verify the snapshot and the restore

Trigger the SLM policy manually to verify it works; the snapshot state should be SUCCESS. Perform restore tests regularly, because an untested backup is not a backup.

Elasticsearch

POST /_slm/policy/daily_backup/_execute
GET /_snapshot/backup_repo/_all
# restore:
POST /_snapshot/backup_repo/<snapshot_name>/_restore

6. Set up remote DR with Cross-Cluster Replication (CCR)

For cross-region disaster recovery, a second cluster (follower) follows indices in the remote primary cluster (leader). First define the remote cluster, then attach the follower index to the leader; changes on the leader flow automatically. CCR is a Platinum (commercial) feature.

Elasticsearch

# define the remote cluster on the follower cluster:
PUT /_cluster/settings
{ "persistent": { "cluster": { "remote": { "primary": { "seeds": ["leader1:9300"] } } } } }
# create the follower index:
PUT /orders-follower/_ccr/follow
{ "remote_cluster": "primary", "leader_index": "orders" }