Oracle RAC day-to-day operations and failure scenarios

Day-to-day operation and incident response for an installed, running Oracle RAC (Grid Infrastructure 19c) cluster: health check, node eviction diagnosis, taking a node down for maintenance, planned shutdown and startup of the whole cluster, adding and removing nodes, service management (preferred/available, AC/TAC), SCAN/VIP/port/interconnect changes, OCR/OLR backup and restore, relocating OCR and voting files. This is not an installation guide (first RAC installation); for patching see RAC rolling patch, for disk group work ASM day-to-day operations. Every scenario follows the order diagnosis → safety net → apply → verification → follow-up.

Topology: Existing 2–8 node RAC (Grid Infrastructure 19c; 23ai/26ai differences in the notes)

Scenarios

• Cluster health check (daily / before maintenance) — Changes nothing: checks the Clusterware stack, resources, OCR/voting, ASM, interconnect, time and recent alerts. Choose it for the daily check, before every maintenance and patch, and as the first thing to do when a complaint comes in.
• A node was evicted from the cluster or rebooted on its own (eviction diagnosis) — Read-only: choose it if a node rebooted unexpectedly or was evicted from the cluster. The Clusterware alert log, ocssd.trc, operating system logs and the misscount/disktimeout values show whether the root cause is the network, the disk or the operating system.
• Stop a node for maintenance and bring it back — Choose it when an operating system patch, kernel or hardware maintenance is planned on a node. The other nodes keep running; the services of the instance on that node are relocated and sessions are closed at the end of the drain timeout.
• Planned shutdown and startup of the whole cluster — Choose it when all nodes must be shut down at the same time, such as for SAN or data center maintenance. Full outage: all databases, ASM and Clusterware stop on all nodes.
• Add a new node to the cluster — Choose it when a new server joins the cluster (capacity increase, hardware refresh). No downtime: pre-checks (cluvfy), Grid home (addnode.sh), DB home and the new instance (dbca -addInstance).
• Remove a node from the cluster — Choose it when a server is permanently removed from the cluster (return, hardware failure). The instance is deleted, the DB and Grid homes are removed from that node and the node is deleted from the cluster; the other nodes stay up.
• Add, modify, relocate or remove a service — Choose it to define the service the application connects to, change its placement (preferred/available) or failover settings (TAF/Application Continuity), or relocate a running service to another instance. The database stays open.
• SCAN, VIP, listener port or interconnect change — Choose it when the SCAN name/IPs, a node VIP, the listener port or the private network (interconnect) will change. An interconnect change requires stopping the whole cluster; the others cause a short connection outage.
• Back up OCR and OLR, check the backups — No downtime: manual OCR backup, OLR backup on every node and copying the backups outside the cluster disks. Choose it before any risky work such as patching, adding/removing nodes or network changes.
• OCR corrupt or CRS disk group lost: restore from backup — Choose it if ocrcheck shows corruption, CRSD does not start (PROT-/PROC- errors) or the CRS disk group is completely lost. The whole cluster stops; the OCR is restored from backup in exclusive mode, and if the disk group is lost, the disk group, voting files and ASM spfile are re-created.
• Clusterware does not start on one node: OLR corrupt — Choose it if, while the other nodes are running, crsctl start crs cannot bring up OHASD on just one node and ocrcheck -local reports an OLR error. Only that node is affected: the OLR is restored from backup.
• Move OCR and voting files to another disk group — No downtime: ocrconfig -add/-delete and crsctl replace votedisk. Choose it for a storage (SAN) replacement, to increase the redundancy of the CRS disk group, or to also write the OCR to a second disk group.

Before you start

• There is root access on all nodes and a login shell for the grid/oracle users (sudo -iu grid); the grid and oracle profiles load the Grid/DB home environment.
• The cluster is healthy before you start: CRS online on all nodes, no resource down that should be up. If not, fix the problem first with the health check.
• The maintenance window and change record are approved; application teams are informed; a blackout is set in the monitoring system. …This operation causes a full outage.…
• There is a current RMAN backup of the database and archived log backups… (adding/deleting an instance changes redo threads and undo tablespaces)….
• There is an OCR backup (…) stored outside the cluster disks, or the path of the automatic backup in an intact disk group is known. …Enough empty disks, visible on all nodes, are ready for … redundancy of the CRS disk group.…
• The new node (…) runs the same operating system release and kernel as the existing nodes; the operating system preparation from RAC installation and storage preparation are done; the public/private interface names are the same; the shared LUNs are presented to this node too; the VIP address … is free.
• The approximate time of the incident is known. If AHF (TFA) is installed, diagnostic collection takes a single command; otherwise the logs are reviewed manually.

Phases of the guide

• Cluster health check — Check the Clusterware stack on all nodes · List resource states · Check database, service and listener status · Check OCR, voting and OLR integrity · Check ASM and disk group usage · Check the network configuration and time synchronization …
• Stop the whole cluster — Record the state before stopping · Take a manual OCR backup (safety net) · Stop all databases · Stop Clusterware on all nodes · Stop OHASD (…) · Do the maintenance work
• Start the whole cluster — Start Clusterware (…) · Wait for all nodes to join the cluster · Verify that the databases are open · Compare resources after startup
• Preparing the new node and pre-checks — Record the current cluster configuration · Verify the new node's operating system preparation · Add the new node to /etc/hosts (…) · Extend SSH user equivalence for the grid user to the new node · Extend SSH user equivalence for the oracle user to the new node · Send the cvuqdisk RPM to the new node …
• Extending Grid Infrastructure to the new node — Take a manual OCR backup (safety net) · Extend the Grid home to the new node (addnode.sh) · Run orainstRoot.sh (new node) · Run root.sh (new node) · Set up the grid environment on the new node · Verify the new node in the cluster
• DB home and new instance — Extend the DB home to the new node · Run the DB home root.sh (new node) · Set up the oracle environment on the new node · Add the new instance (dbca -addInstance) · Add the new instance to the service configuration · Verify the new instance
• After adding the node — Back up the new node's OLR · Back up the OCR of the new configuration · Update monitoring, backup and operations jobs
• Removing the instance and the DB home — Drain the services from … (the instance stays open) · Delete the instance (dbca -deleteInstance) · Narrow the DB home inventory to the target node (…) · Remove the DB home (deinstall -local) (…) · Update the DB home inventory on the remaining nodes
• Removing Clusterware from the target node — Check the pin state and unpin if needed · Remove the Grid home (deinstall -local) (…) · Delete the node from the cluster definition · Verify the deletion and remove the leftover VIP
• After removing the node — Back up the OCR of the new configuration · Clean up leftovers and update records
• Service: diagnosis — Show the current service configuration
• Service: apply — Add and start the service · Change the service placement and settings · Start the service according to the new placement · Relocate the service to another instance · Stop and remove the service
• Service: verification — Verify that the service is visible in the database and the listener
• Network change: diagnosis and safety net — Record the current network configuration · Take a manual OCR backup (safety net) · Back up the GPnP profile (…)
• Change the SCAN name / IPs — Verify the new SCAN resolution · Re-create the SCAN VIPs and listeners with the new values · Update the database's remote_listener setting · Verify connections through SCAN
• OCR and OLR backup — Show the existing OCR backups and the backup location · Take a manual OCR backup · Copy the latest manual backup outside the cluster and verify it · Take a manual OLR backup (…) · Also back up the ASM spfile and password file
• OCR restore: diagnosis — Check the OCR state and Clusterware · Check the disk group and disks from the ASM side · Verify the backup to restore
• OCR restore: apply — Stop Clusterware (…) · Start Clusterware in exclusive mode on node 1 · Start ASM with a temporary parameter file (if the spfile is lost) · Re-create the … disk group · Restore the OCR from backup · Re-create the voting files …
• OCR restore: verification and follow-up — Verify the cluster and the OCR · Check for changes made after the backup · Take a new OCR backup and fix the root cause