MongoDB backup and restore (mongodump, mongorestore, snapshot)

Sets up a backup routine for an installed MongoDB and walks through restoring from backup scenario by scenario. Backup: mongodump of a single collection, a database or the whole deployment (--archive --gzip; on a replica set a consistent point with --oplog, reading from a secondary), credentials file, scheduled backups with retention via systemd timer/cron, test restore and an optional file system snapshot. Restore: restore rehearsal, full restore, single database/collection, point-in-time recovery from backup + oplog, clone to another cluster, and sharded cluster. For hundreds of GB of data, when fast recovery is required, or on a sharded cluster, snapshots or an enterprise tool (PBM, Ops Manager) are preferred over mongodump; the choice is explained in the first steps.

Topology: Backup server or a member + replica set / single instance / sharded cluster (mongos)

Scenarios

• Set up backups: mongodump, scheduling and test restore — Backup user, credentials file, manual and scheduled mongodump, test restore under a new name; optional snapshot and mongoexport. Start here if you are setting this up for the first time.
• Can the backup really be restored? (--dryRun and rehearsal on a temporary instance) — Without touching live data: archive integrity, the namespaces and oplog it contains, and optionally a full restore to a temporary mongod with document counts.
• Restore all data from backup (full restore, --drop) — Data has been largely lost or corrupted and the deployment will go back to the moment the backup was taken. Writes made after the backup are lost; the application is down for the duration of the restore.
• Bring back a single database or collection (--nsInclude, under a new name or in place) — A collection/database was dropped or corrupted, the rest is intact: only that namespace comes from the backup; other data is untouched and the application can stay up.
• Return to a specific moment: backup + oplog (accidental drop, bad update/delete) — The bad operation happened after the backup and the writes up to that moment are also needed: the last full backup is restored and the live oplog is applied up to just before the bad operation (--oplogReplay --oplogLimit). Replica set only.
• Restore the backup to another cluster / server (clone, test environment, migration) — The source deployment is not touched: the backup is restored to another replica set or instance (test/reporting copy, server migration, version upgrade); users are optionally migrated.
• Restore a sharded cluster: the limits of mongodump and the right method — The backed-up topology is a sharded cluster: logical restore through mongos (recreating shard keys, balancer) or physical rebuild of the whole cluster from coordinated snapshots.

Before you start

• The server being backed up runs a supported operating system; the clock is synchronized with NTP and the time zone is correct (backup file names and scheduling use local time).
• This server can resolve the names of and reach the ports on ….… On a replica set the names in the connection string must match the member names in rs.conf(); the driver connects to those names.…
• … is on a disk separate from the data disk or on a network share, and its free space is at least the compressed backup size (usually 20–50% of the data) times the retention period. A backup on the same disk is lost together with the data if the disk is lost.
• Authorization is enabled and you have the … user (root, or the userAdminAnyDatabase + clusterAdmin roles) and its password. The backup user is created with this user.
• The oplog window is clearly longer than the duration of a full backup (at least several hours). --oplog collects the oplog entries generated during the backup; if the window is shorter than the backup, the backup fails.… For point-in-time recovery the first entry in the live oplog must be earlier than the moment the full backup to be used finished; otherwise the operations in between exist nowhere.…
• There is HTTPS access to the MongoDB repo (repo.mongodb.org) or an internal mirror is defined; if the tools are already installed on the server, their versions are shown.
• dbPath (…) is on the … LV and the volume group has at least … free. The WiredTiger journal (…/journal) must be on the same LV; if it was moved to another disk, fsyncLock before the snapshot is mandatory.
• You have root or sudo privileges. If a full restore will be done, a maintenance window and the application team's approval have been obtained.… Before the restore, the incident has been recorded (when, which operation, who); the application team knows which data will come back.…
• Backups are set up with this guide's "Set up backups" scenario: the tools (mongorestore, mongosh) and the credentials file … (…: backup + restore) exist on this server. If not, first run the first two phases of the Set up backups scenario.
• A restore rewrites data and indexes: the data disk of the target servers has free space of at least 1.5 times the (uncompressed) data being restored.… On a replica set the secondaries receive the same writes through replication; the oplog window shrinks during the restore.…
• … has free space equal to the uncompressed size of the data being restored (full backup + oplog). The temporary mongod listens only on 127.0.0.1 on port …; the port must be free.
• The target (…) is installed and running, authorization is enabled, and you have the password of the … (root) user. The target version is the same as the source or newer (e.g. 7.0 → 8.0); restoring to an older version is not supported.

Phases of the guide

• Installing the tools — Add the MongoDB … repo · Add the MongoDB … repo · Install mongodump, mongorestore and mongosh · Prepare the backup directory
• Backup user and credentials file — Create the backup user · Write the credentials file (--config) · Test the connection
• Manual backup (mongodump) — Check the data size and the oplog window · Stop the balancer · Take a backup with mongodump · Restart the balancer · Verify the backup (gzip and dryRun) · Subset backup with a query (--query)
• Restore (mongorestore) — Test restore under a new name (--nsFrom/--nsTo) · Compare document counts · Drop the test database · Define a role for oplogReplay · In-place full restore (--drop) · Revoke the temporary privilege …
• mongoexport / mongoimport (data transfer, not a backup) — Export the collection (mongoexport) · Import into a separate collection (mongoimport) · Compare the types and clean up
• File system snapshot — Take an LVM snapshot (with fsyncLock) · Archive the snapshot to the backup directory · Take a file copy with fsyncLock
• Sharded cluster backup: limits and the enterprise path — Choosing a method for a consistent sharded cluster backup
• Scheduled backup and retention — Restrict the balancer to run outside the backup hours · Write the backup script · Make the script executable · Prepare SSH access to the remote server · Create the systemd service and timer · Trigger the scheduled backup by hand and verify it …
• Monitoring and maintenance — Monitor the age of the last backup · Regular recovery drills
• 1. Diagnosis: backup files and tools — List the backup files and the latest backups in order
• 2. Rehearsal: reading the archive — Check compression integrity (gzip -t) · Try the restore without writing data (--dryRun) · Does the archive contain an oplog? (consistent point)
• 3. Rehearsal: full restore to a temporary instance — Prepare mongod for the temporary instance · Start the temporary mongod · Restore the backup to the temporary instance · Count the restored data · Stop and remove the temporary instance
• 4. Afterwards — Record the result of the rehearsal
• Sharded cluster: a full restore is not done with this scenario — Go to the sharded cluster scenario
• 1. Diagnosis: is a full restore the right scenario? — Look at the current state and the affected data · Try the restore without writing data (--dryRun)
• 2. Safety net — Stop the application and check the connections · Back up the current state (fallback point)
• 3. Execution: full restore — Grant a temporary privilege for oplogReplay · Restore the backup in place (--drop… --oplogReplay…) · Revoke the temporary privilege · Clean up the restore's temporary user collections
• 4. Verification — Check the document and index counts · Check that the replica set has caught up
• 5. Afterwards — Take a new full backup and start the application
• 1. Diagnosis: the current state of the namespace and its state in the backup — Look at the live state · Try the restore without writing data (--dryRun)
• 2. Safety net — Back up the current state (fallback point)
• 2. Execution: restore under a new name — Restore the backup under a new name (--nsFrom/--nsTo) · Compare the counts in the backup and in live · Add the deleted documents back to live ($merge) · Drop the temporary copy
• 3. Execution: restore in place (--drop) — Preserve the shard key: reshard the collection · Restore the namespace in place
• Point-in-time recovery only on a replica set — Point-in-time recovery with the oplog is not possible on this deployment
• 1. Diagnosis: the moment of the bad operation and the oplog window — Find the bad operation in the oplog · Verify that the full backup is within the oplog window
• 2. Safety net: copy the live oplog — Take a copy of the live oplog · Back up the current state (fallback point)
• 3. Execution: go back to that moment on a temporary instance — Prepare mongod for the temporary instance · Start the temporary mongod · Restore the full backup to the temporary instance · Apply the live oplog up to just before the bad operation · Check the recovered data on the temporary instance · Copy the recovered data to live …
• 4. Afterwards — Take a new full backup and fix the root cause
• 1. Diagnosis: is the target right and compatible? — Verify that the target is not the source and check its version · Try the restore without writing data (--dryRun)
• 2. Safety net and target credentials — Write a temporary credentials file for the target · Back up the existing data on the target