During the Oracle Grid Infrastructure upgrade from 19c to 26ai, the upgrade failed on rac1 while running:
/u01/app/23.0.0/grid/rootupgrade.sh
The failure occurred at:
2026/08/14 02:43:14 CLSRSC-595: Executing upgrade step 14 of 16: 'UpgradeCluster'.
After this point, rac1 entered a reboot loop. CRS autostart was disabled to stop the continuous reboots.
Later, starting Clusterware manually failed:
[root@rac1 trace]# crsctl start crs -wait
CRS-6705: Oracle Clusterware Release Version ('19.0.0.0.0') does not match Software Version ('23.0.0.0.0'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
The cluster was left in a partially upgraded state: the cluster active version was still 19c, while rac1 had already been partially switched to the 26ai software.
1. Verify the cluster upgrade state
On the healthy node, rac2:
/u01/app/19.3.0/grid/bin/crsctl query crs activeversion -f
Output:
Oracle Clusterware active version on the cluster is [19.0.0.0.0].
The cluster upgrade state is [ROLLING UPGRADE].
The cluster active patch level is [1191252804].
This confirmed that the cluster active version had not yet been advanced to 26ai.
2. Restore rac1 configuration in OCR
From rac2, where the 19c Clusterware stack was still running:
/u01/app/19.3.0/grid/bin/clscfg -nodedowngrade-h rac1
Output:
clscfg: -nodedowngrade mode specified
clscfg: EXISTING configuration version 19 detected.
Successfully restored the node specific configuration from backup for host:[rac1]
This restores the pre-upgrade cluster metadata associated with rac1 in OCR as part of terminating the incomplete upgrade.
3. Downgrade the local Clusterware configuration on rac1
On rac1 as root, run the downgrade from the 26ai Grid home:
cd /u01/app/23.0.0/grid/crs/install./rootcrs.sh -downgrade -online
Output:
2026/08/14 04:38:58 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2026/08/14 04:38:59 CLSRSC-592: Run 'crsctl start crs' from home /u01/app/19.3.0/grid on each node to complete downgrade.
This reverts the local Clusterware configuration on rac1 to the previous GI release.
4. Start CRS from the 19c Grid home
On rac1:
/u01/app/19.3.0/grid/bin/crsctl start crs -wait
Clusterware now started successfully:
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded...
CRS-4123: Oracle High Availability Services has been started.
5. Verify both nodes are back on 19c
/u01/app/19.3.0/grid/bin/crsctl check cluster -all
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac1
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac2
Both nodes returned:
19.0.0.0.0
However:
/u01/app/19.3.0/grid/bin/crsctl query crs activeversion -f
still showed:
Oracle Clusterware active version on the cluster is [19.0.0.0.0].
The cluster upgrade state is [ROLLING UPGRADE].
The rollback was therefore not yet complete.
6. Stop ASM rolling migration
As grid, connect to ASM:
sqlplus / as sysasm
Run:
ALTER SYSTEM STOP ROLLING MIGRATION;
7. Start the formal failed-upgrade downgrade
From the 26ai Grid home on rac1, as grid:
/u01/app/23.0.0/grid/gridSetup.sh -silent -downgrade -nodes rac1 -oldHome /u01/app/19.3.0/grid
In this case, gridSetup.sh generated the following root action:
/u01/app/23.0.0/grid/crs/install/rootcrs.sh -downgrade -auto
8. Stop CRS on the other node before the last-node operation
The first attempt at rootcrs.sh -downgrade -auto failed because rac2 was still running:
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac2, number 2, and is terminating
CLSRSC-549: The root script cannot downgrade this node rac1 with last node operations because the Grid Infrastructure stack is up on at least one other cluster node.
Because rac1 was the node being processed with last-node downgrade operations, the GI stack on the other node had to be stopped temporarily.
On rac2 as root:
/u01/app/19.3.0/grid/bin/crsctl stop crs
9. Complete the last-node downgrade on rac1
On rac1 as root:
/u01/app/23.0.0/grid/crs/install/rootcrs.sh -downgrade -auto
This time the command can perform the required last-node operations because no other CSS daemon is active in the cluster.
10. Start the 19c Clusterware stack
After the downgrade succeeds, start rac1 explicitly from the 19c Grid home:
/u01/app/19.3.0/grid/bin/crsctl start crs -wait
Then restart the 19c stack on rac2, which was stopped for the last-node downgrade:
/u01/app/19.3.0/grid/bin/crsctl start crs -wait
11. Update Oracle Inventory on a failed node
Run the commands generated by gridSetup.sh as the grid owner.
Mark the failed 26ai Grid home as no longer being the CRS home:
$ /u01/app/23.0.0/grid/oui/bin/runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/23.0.0/grid "CLUSTER_NODES=rac1" -doNotUpdateNodeList
Then register the 19c Grid home as the CRS home again:
$ /u01/app/23.0.0/grid/oui/bin/runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/19.3.0/grid "CLUSTER_NODES=rac1" -doNotUpdateNodeList
These are not manually invented cleanup commands; they are the exact inventory commands produced by gridSetup.sh -downgrade for this failed upgrade.
12. Verify that the rollback is fully complete
Check both nodes:
/u01/app/19.3.0/grid/bin/crsctl check cluster -all
Verify their software versions:
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac1
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac2
Both should report:
19.0.0.0.0
Finally:
/u01/app/19.3.0/grid/bin/crsctl query crs activeversion -f
The rollback is complete when the cluster reports:
Oracle Clusterware active version on the cluster is [19.0.0.0.0].The cluster upgrade state is [NORMAL].
Only after reaching NORMAL should another GI upgrade attempt be started.
DBA Knowledge Base 