Oracle GoldenGate day-to-day operations and failure scenarios (Microservices 21c / 23ai)

Day-to-day operations and failure recovery runbook for an installed GoldenGate Microservices replication (for installation see the GoldenGate installation guide). If you are not sure what to do, start with 'Health and lag check': the read-only diagnosis points you to the right scenario — Extract and Replicat ABEND (report, discard, OGG-xxxxx errors), adding a populated table to replication (consistent SCN + AFTERCSN), removing a table, resyncing a single table, trail cleanup (Purge Trail), repositioning Extract from an SCN, version upgrade/patch, and keeping Extract running after a Data Guard switchover/failover on the source. Every scenario follows the pattern diagnosis → safety net → apply → verification.

Topology: Source server (source deployment, Integrated Extract, Distribution Path) + target server (target deployment, Parallel Replicat)

Scenarios

• Health and lag check (is everything all right?) — Read-only: Service Manager and the microservices, INFO ALL, lag, STATS, heartbeat lag, capture status in the database and ggserr.log; nothing is changed. If you are not sure what to do, start here.
• Extract ABENDED (stopped with an error) — The Extract on the source is ABENDED or keeps restarting: the cause (connection, missing archived log, log mining, privileges, disk) is found from the OGG-xxxxx error in the report file, fixed, and the Extract is restarted from where it stopped. The source database is not affected; lag builds up.
• Replicat ABENDED (cannot apply to the target) — The Replicat on the target could not apply a transaction: the table, operation and error are found from the report and discard files; the data is fixed or temporarily skipped with REPERROR, and the Replicat is restarted from where it stopped. The target stays open; lag builds up.
• Add a populated table to replication (initial load + AFTERCSN) — Adds an existing table that already contains data to a running replication without downtime: ADD TRANDATA, expdp/impdp at a consistent SCN, and a separate Replicat that applies only this table's changes after that SCN (AFTERCSN). The other tables keep flowing.
• Remove a table from replication — The table will no longer be replicated: TABLEEXCLUDE in the Extract (or removing the TABLE line), MAPEXCLUDE in the Replicat, and DELETE TRANDATA if needed. The processes are restarted briefly.
• Resynchronize a single table (resync) — A table's data on the target has diverged from the source (manual change, skipped error, incomplete initial load): the table is reloaded from the source at a consistent SCN and a temporary Replicat applies the changes after that SCN. The other tables are not affected.
• Trail files are filling the disk: cleanup (Purge Trail) — The trail directory is growing or the disk is about to fill up: you see which files are being read, a Purge Trail task (the equivalent of PURGEOLDEXTRACTS in Classic) is defined with USECHECKPOINTS; in an emergency, files already read are deleted manually.
• Reposition Extract to a specific SCN — There is an SCN from which you want Extract to start over (trail loss, wrong starting point, rollback on the target): it is started from a new trail file with ALTER EXTRACT … SCN and ETROLLOVER. The same data may arrive at the target a second time.
• GoldenGate version upgrade / patch (Microservices) — Moving to a new GoldenGate release or patch: out-of-place (install a new home, move Service Manager and the deployments to the new home with REST) or in-place with OPatch. Extract/Replicat stop briefly; rollback means going back to the old home.
• Keep Extract running after a Data Guard switchover/failover on the source — A switchover or failover happened on the source database; Extract must connect to the new primary and continue from its checkpoint: the connection (credential alias/service), capture status, HANDLEDLFAILOVER and possible data divergence are checked.

Before you start

• GoldenGate Microservices installed as in the installation guide: on the source, Integrated Extract … and Distribution Path … in deployment …; on the target, Parallel Replicat … in deployment …; credential aliases ggsrc/ggsrc_pdb (source) and ggtgt (target).
• You have the oggadmin password of both deployments (the steps prompt for it; it is not written into scripts).… For the initial load, the SYSTEM password in both PDBs and the OGG_DP directory (…) from the installation are ready.…
• The Service Manager systemd service (OracleGoldenGate) is running on both servers.
• A change record/maintenance window is open: …Extract, Distribution Path and Replicat are stopped during the upgrade; lag builds up and drains when the work is done…when Extract is rewound the same transactions may arrive at the target a second time; a collision plan has been agreed with the application owner…operations skipped with REPERROR will be missing on the target; the business owner has approved…the processes are restarted briefly….

Phases of the guide

• Status: processes, lag and services (read-only) — Check Service Manager and the microservice ports · Show the source processes, Extract lag and the Distribution Path (source) · Show the target processes and Replicat lag (target) · Measure end-to-end lag with the heartbeat (target) · Check the capture status and required archived logs in the source database (source) · Scan the error log (source) …
• 1. Diagnosis: why did Extract stop? — Show the Extract status and the error in the report (source) · Check the capture status and archived logs in the database (source)
• 2. Safety net — Back up the parameter, checkpoint and report files (source)
• 3. Apply: fix the cause and start Extract — Restore the missing archived logs from the RMAN backup (source) · Fix the cause · Start Extract and monitor it (source)
• 4. Verification and follow-up — Verify that the lag drains (source) · Prevent the root cause (source)
• 1. Diagnosis: at which operation did Replicat stop? — Show the Replicat status, the report and the discard file (target) · Compare the failing row on the target and the source
• 2. Safety net — Back up the parameter, checkpoint and report files (target)
• 3. Apply: fix the data and start Replicat — Fix the data on the target · Temporarily redirect the error to the discard file: REPERROR (target) · Start Replicat (target)
• 4. Verification and follow-up — Verify that Replicat is advancing (target) · Remove the temporary REPERROR (target)
• 1. Diagnosis: is the table suitable for replication? — Inspect the table on the source (source) · Show the supplemental logging status (source) · Check the table on the target (target)
• 2. Safety net — Back up the parameter and checkpoint files (source) · Back up the parameter and checkpoint files (target) · Back up the target table (target)
• 3. Apply: load the table and start it with its own Replicat — Enable supplemental logging at table level (source) · Exclude the table from the main Replicat: MAPEXCLUDE (target) · Remove the table's explicit MAP line from the main Replicat (target) · Add the table to the Extract (source) · Add a new Replicat for this table only (target) · Export the table at a consistent SCN (source) …
• 4. Verification and follow-up — Compare row counts (source) · Check the row count and the new Replicat (target)
• 1. Diagnosis: where is the table defined? — Show where the table appears in the parameter files and its supplemental logging (source) · Show the table's mapping in the Replicat (target)
• 2. Safety net — Back up the parameter and checkpoint files (source) · Back up the parameter and checkpoint files (target)
• 3. Apply: remove the table from Extract and Replicat — Remove the table from the Extract (source) · Wait for the remaining trail records to be applied (target) · Remove the table from the Replicat (target) · Remove the table's MAP line (target) · Remove table-level supplemental logging (source)
• 4. Verification — Verify that the table is no longer processed (source)
• 1. Diagnosis: how much space do the trails use, and who reads them? — Show the trail directory and files (source) · Show the trail directory and the position Replicat reads (target)
• 2. Apply: define the Purge Trail task — Define the Purge Trail task in the source deployment · Define the Purge Trail task in the target deployment
• 3. Emergency: delete read trail files manually — Delete the read trail files
• 1. Diagnosis: can we go back to the SCN? — Show the current position of the Extract (source) · Check the capture's first SCN and the archived logs (source)
• 2. Safety net — Back up the parameter, checkpoint and report files (source)
• 3. Apply: position Extract to the SCN — Stop Extract, position it to the SCN and start it from a new trail file (source)
• 4. Verification — Verify that Extract is advancing from the new position (source) · Watch for collisions on the target (target)
• 1. Diagnosis: current version and preparation — Record the current version, patches and processes (source) · Record the current version, patches and processes (target)
• 2. Safety net — Back up the deployment and Service Manager directories (source) · Back up the deployment and Service Manager directories (target)
• 3. Apply: stop the processes and upgrade — Stop the processes cleanly (source) · Stop the processes cleanly (target) · Install the new GoldenGate home (source) · Install the new GoldenGate home (target) · Move Service Manager and the deployment to the new home (source) · Apply the patch with OPatch (source) …
• 4. Verification — Start the processes and verify the version (source) · Start the processes and verify the version (target) · Rollback (if needed)
• 1. Diagnosis: the new primary and the Extract's connection — Check the role and capture status of the new primary (new primary) · Show the Extract status and the credential aliases (source)
• 2. Safety net — Back up the parameter, checkpoint and report files (source)