Diagnose the cause of a node eviction / unexpected node reboot

Command

# 1) When, which node, evicted by whom: CRS alert log (on every node)
grep -E "CRS-1601|CRS-1607|CRS-1608|CRS-1609|CRS-1610|CRS-1611|CRS-1612|CRS-8011" \
  $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/alert.log
# 2) CSS details: was the network heartbeat or the disk heartbeat lost
grep -iE "evict|misscount|heartbeat|disk HB|network HB" \
  $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/ocssd.trc | tail -100
# 3) Who triggered the reboot (cssdagent/cssdmonitor):
ls -lt $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/ohasd_cssd*.trc
# 4) OS resources at the time of the event (Cluster Health Monitor):
oclumon dumpnodeview -allnodes -s "2026-01-15 03:10:00" -e "2026-01-15 03:20:00"
# 5) Diagnostic collection from all nodes (AHF/TFA):
tfactl diagcollect -crs -os -last 4h
# 6) OS side: /var/log/messages, journalctl -k (OOM, NIC, multipath, I/O errors)

What it does

Follows the node eviction chain in order: CRS-16xx messages in the CRS alert log, heartbeat losses in ocssd.trc, OS metrics from CHM, and the logs of all nodes collected with TFA.

When to use it

When a node reboots unexpectedly or is evicted from the cluster, for root cause analysis (RCA) and an Oracle Support SR.

Caution

Collect logs from every node and covering the event time; trace files rotate and evidence can be lost. Verify the time zones are the same across nodes (CTSS/NTP). Do not apply "fixes" such as raising misscount before evidence is collected.