How to Change the Domain Name of an Oracle RAC Cluster

Changing the domain name of an existing Oracle RAC cluster requires more than changing the hostname at the operating-system level. The old domain may be referenced in DNS, operating-system configuration, Oracle Net files, VIP configuration, and SCAN resources.

This article describes the steps for changing the domain name of a two-node Oracle RAC cluster.

Example

Assume the existing nodes are:

rac1.olddomain.com
rac2.olddomain.com

and they need to be changed to:

rac1.newdomain.com
rac2.newdomain.com

The SCAN name changes from rac-scan.olddomain.com to rac-scan.newdomain.com.


1. Stop the databases

Before making the domain-name changes, stop all database instances running (if any) on the cluster.

For example:

srvctl stop database -d <DB_UNIQUE_NAME>

Verify:

srvctl status database -d <DB_UNIQUE_NAME>

2. Update DNS

Update the DNS records to reflect the new domain name.

Review the records for:

rac1.newdomain.com
rac2.newdomain.com
rac1-vip.newdomain.com
rac2-vip.newdomain.com
rac-scan.newdomain.com

Make sure the IP addresses themselves remain mapped correctly.

For SCAN, verify that the new SCAN name resolves to all configured SCAN IP addresses.

For example:

nslookup rac-scan.newdomain.com

or:

dig rac-scan.newdomain.com

3. Update the operating-system configuration

Review the operating-system files that may contain the old domain.

Depending on the Linux distribution and configuration, this may include:

/etc/hosts
/etc/resolv.conf
/etc/sysconfig/network
/etc/sysctl.conf

For example, /etc/resolv.conf may contain:

search olddomain.com

which should be changed to:

search newdomain.com

Similarly, /etc/sysconfig/network may contain:

DOMAINNAME=olddomain.com

and /etc/sysctl.conf may contain:

kernel.domainname=olddomain.com

4. Review Oracle Net configuration

Check the Oracle Grid Infrastructure and Database Oracle homes for references to the old domain.

Review files such as:

<GI_HOME>/network/admin/tnsnames.ora
<DB_HOME>/network/admin/tnsnames.ora
<GI_HOME>/network/admin/listener.ora
<DB_HOME>/network/admin/sqlnet.ora

Search for the old domain:

grep -Ri "olddomain.com" <GI_HOME>/network/admin
grep -Ri "olddomain.com" <DB_HOME>/network/admin

Check for IFILE

tnsnames.ora may not contain the actual entry directly. It can reference another Oracle Net configuration file using IFILE.

For example:

IFILE=/path/to/additional_tnsnames.ora

Check for it with:

grep -i '^[[:space:]]*IFILE' <GI_HOME>/network/admin/tnsnames.ora
grep -i '^[[:space:]]*IFILE' <DB_HOME>/network/admin/tnsnames.ora

If an IFILE is configured, review the referenced file for the old domain as well.


5. Change the operating-system hostname

Check the current hostname:

hostname
hostname -f
hostname -s
hostnamectl

For example:

hostname -f
rac1.olddomain.com
hostname -s
rac1

Change the hostname to the new FQDN:

hostnamectl set-hostname rac1.newdomain.com

Repeat the operation on the second node:

hostnamectl set-hostname rac2.newdomain.com

Verify:

hostname
hostname -f
hostname -s

The expected result should be similar to:

hostname -f
rac1.newdomain.com
hostname -s
rac1

The FQDN changes, while the RAC node short name remains the same.


6. Check the VIP configuration before modifying it

Oracle’s procedure includes changing the VIP when the VIP resource itself contains the old domain name.

However, this step may not be necessary in every RAC configuration.

Check the existing VIP configuration first:

<GI_HOME>/bin/srvctl config vip -node rac1
<GI_HOME>/bin/srvctl config vip -node rac2

For example, if the output is:

VIP exists: network number 1, hosting node rac1
VIP Name: rac1-vip
VIP IPv4 Address: 192.168.1.101

Then Clusterware stores only the short VIP name. In this case, there is no old domain name stored in the VIP resource, and the VIP modification step can be skipped.

However, if the configuration shows:

VIP Name: rac1-vip.olddomain.com

stop the VIP and modify it to use the new domain:

# As GI owner
<GI_HOME>/bin/srvctl stop vip -node rac1 -force
# As root
<GI_HOME>/bin/srvctl modify vip \
-node rac1 \
-address rac1-vip.newdomain.com/255.255.255.0/<public_interface> \
-netnum 1

Then start and verify the VIP:

# As GI owner
<GI_HOME>/bin/srvctl start vip -node rac1
<GI_HOME>/bin/srvctl config vip -node rac1
<GI_HOME>/bin/srvctl status vip -node rac1

Repeat the same steps for the second node:

# As GI owner
<GI_HOME>/bin/srvctl stop vip -node rac2 -force
# As root
<GI_HOME>/bin/srvctl modify vip \
-node rac2 \
-address rac2-vip.newdomain.com/255.255.255.0/<public_interface> \
-netnum 1
# As GI owner
<GI_HOME>/bin/srvctl start vip -node rac2
<GI_HOME>/bin/srvctl config vip -node rac2
<GI_HOME>/bin/srvctl status vip -node rac2

The important point is: do not modify the VIP blindly. First check whether the Clusterware VIP resource actually contains the old domain name.


7. Check the current SCAN configuration

Check SCAN:

<GI_HOME>/bin/srvctl config scan

For example:

SCAN name: rac-scan.olddomain.com, Network: 1

Unlike the VIP configuration, SCAN commonly contains the full domain name.


8. Stop the SCAN listeners and SCAN

As the Grid Infrastructure owner:

<GI_HOME>/bin/srvctl stop scan_listener
<GI_HOME>/bin/srvctl stop scan

Verify that the resources are stopped if necessary:

<GI_HOME>/bin/srvctl status scan
<GI_HOME>/bin/srvctl status scan_listener

9. Change the SCAN name

After the DNS records for the new SCAN name are configured, modify SCAN.

Run the modification from root user:

<GI_HOME>/bin/srvctl modify scan -n rac-scan.newdomain.com

Verify the new configuration:

<GI_HOME>/bin/srvctl config scan

Expected:

SCAN name: rac-scan.newdomain.com, Network: 1

10. Update the SCAN listeners

Update the SCAN listener configuration:

<GI_HOME>/bin/srvctl modify scan_listener -u

Start the SCAN listeners:

<GI_HOME>/bin/srvctl start scan_listener

Then verify:

<GI_HOME>/bin/srvctl status scan
<GI_HOME>/bin/srvctl status scan_listener

All SCAN VIPs and SCAN listeners should be online.


11. Stop Oracle Clusterware

Once the required changes have been completed, stop Clusterware on all nodes:

# As root
<GI_HOME>/bin/crsctl stop cluster -all

Verify that Clusterware shuts down cleanly.


12. Restart the RAC nodes

Restart all RAC nodes using the normal operating-system procedure:

reboot

After the nodes return, verify the hostname:

hostname
hostname -f

Expected:

rac1.newdomain.com

Repeat the validation on every RAC node.


13. Validate node name resolution

Check the node names:

getent hosts rac1
getent hosts rac2

Expected:

192.168.1.1 rac1.newdomain.com
192.168.1.2 rac2.newdomain.com

14. Validate VIP resolution

Check the VIP names:

getent hosts rac1-vip
getent hosts rac2-vip

Expected:

192.168.1.101 rac1-vip.newdomain.com
192.168.1.102 rac2-vip.newdomain.com

Even when Clusterware stores the VIP as only rac1-vip, the operating system should resolve that short name correctly.


15. Validate SCAN resolution

First retrieve the exact SCAN configured in Clusterware:

srvctl config scan

Then validate that exact name:

getent hosts rac-scan.newdomain.com

or:

dig rac-scan.newdomain.com

For a three-SCAN-VIP configuration, all three expected IP addresses should resolve.

Example:

192.168.1.201 rac-scan.newdomain.com
192.168.1.202 rac-scan.newdomain.com
192.168.1.203 rac-scan.newdomain.com

16. Validate RAC node membership

Run:

olsnodes

The RAC node names should normally remain the same short names:

rac1
rac2

This was also the result after the domain change in the tested cluster.


17. Validate Clusterware resources

Run:

crsctl status resource -t

Verify that the important RAC resources are online, including:

ora.LISTENER.lsnr
ora.asm
ora.rac1.vip
ora.rac2.vip
ora.scan1.vip
ora.scan2.vip
ora.scan3.vip
ora.LISTENER_SCAN1.lsnr
ora.LISTENER_SCAN2.lsnr
ora.LISTENER_SCAN3.lsnr

After the tested domain change and restart, the node VIPs, three SCAN VIPs, SCAN listeners, ASM, and other Clusterware resources returned to ONLINE state.


18. Validate nodeapps and SCAN

Run:

srvctl config nodeapps -a
srvctl config scan

Confirm:

  • the node VIPs have the expected names and IP addresses
  • the SCAN uses the new domain
  • all expected SCAN VIP addresses are present

For example:

VIP Name: rac1-vip
VIP IPv4 Address: 192.168.1.101
VIP Name: rac2-vip
VIP IPv4 Address: 192.168.1.102

and:

SCAN name: rac-scan.newdomain.com

19. Run Cluster Verification Utility

Finally, run the appropriate Cluster Verification Utility checks.

For example:

cluvfy stage -post crsinst -n rac1,rac2

or the appropriate cluvfy command for your Grid Infrastructure version and environment.

Validate:

  • hostname resolution
  • node connectivity
  • public network
  • private interconnect
  • VIPs
  • SCAN
  • SCAN listeners
  • Clusterware
  • ASM
  • database connectivity

Also search the system and Oracle configuration for any remaining references to the old domain:

grep -Ri "olddomain.com" /etc 2>/dev/null
grep -Ri "olddomain.com" <GI_HOME>/network/admin 2>/dev/null
grep -Ri "olddomain.com" <DB_HOME>/network/admin 2>/dev/null

Review the results carefully rather than automatically replacing every occurrence.

Recovering failed GI 19c to 26ai upgrade

During the Oracle Grid Infrastructure upgrade from 19c to 26ai, the upgrade failed on rac1 while running:

/u01/app/23.0.0/grid/rootupgrade.sh

The failure occurred at:

2026/08/14 02:43:14 CLSRSC-595: Executing upgrade step 14 of 16: 'UpgradeCluster'.

After this point, rac1 entered a reboot loop. CRS autostart was disabled to stop the continuous reboots.

Later, starting Clusterware manually failed:

[root@rac1 trace]# crsctl start crs -wait
CRS-6705: Oracle Clusterware Release Version ('19.0.0.0.0') does not match Software Version ('23.0.0.0.0'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

The cluster was left in a partially upgraded state: the cluster active version was still 19c, while rac1 had already been partially switched to the 26ai software.

1. Verify the cluster upgrade state

On the healthy node, rac2:

/u01/app/19.3.0/grid/bin/crsctl query crs activeversion -f

Output:

Oracle Clusterware active version on the cluster is [19.0.0.0.0].
The cluster upgrade state is [ROLLING UPGRADE].
The cluster active patch level is [1191252804].

This confirmed that the cluster active version had not yet been advanced to 26ai.

2. Restore rac1 configuration in OCR

From rac2, where the 19c Clusterware stack was still running:

/u01/app/19.3.0/grid/bin/clscfg -nodedowngrade-h rac1

Output:

clscfg: -nodedowngrade mode specified
clscfg: EXISTING configuration version 19 detected.
Successfully restored the node specific configuration from backup for host:[rac1]

This restores the pre-upgrade cluster metadata associated with rac1 in OCR as part of terminating the incomplete upgrade.

3. Downgrade the local Clusterware configuration on rac1

On rac1 as root, run the downgrade from the 26ai Grid home:

cd /u01/app/23.0.0/grid/crs/install./rootcrs.sh -downgrade -online

Output:

2026/08/14 04:38:58 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2026/08/14 04:38:59 CLSRSC-592: Run 'crsctl start crs' from home /u01/app/19.3.0/grid on each node to complete downgrade.

This reverts the local Clusterware configuration on rac1 to the previous GI release.

4. Start CRS from the 19c Grid home

On rac1:

/u01/app/19.3.0/grid/bin/crsctl start crs -wait

Clusterware now started successfully:

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded...
CRS-4123: Oracle High Availability Services has been started.

5. Verify both nodes are back on 19c

/u01/app/19.3.0/grid/bin/crsctl check cluster -all
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac1
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac2

Both nodes returned:

19.0.0.0.0

However:

/u01/app/19.3.0/grid/bin/crsctl query crs activeversion -f

still showed:

Oracle Clusterware active version on the cluster is [19.0.0.0.0].
The cluster upgrade state is [ROLLING UPGRADE].

The rollback was therefore not yet complete.

6. Stop ASM rolling migration

As grid, connect to ASM:

sqlplus / as sysasm

Run:

ALTER SYSTEM STOP ROLLING MIGRATION;

7. Start the formal failed-upgrade downgrade

From the 26ai Grid home on rac1, as grid:

/u01/app/23.0.0/grid/gridSetup.sh -silent -downgrade -nodes rac1 -oldHome /u01/app/19.3.0/grid

In this case, gridSetup.sh generated the following root action:

/u01/app/23.0.0/grid/crs/install/rootcrs.sh -downgrade -auto

8. Stop CRS on the other node before the last-node operation

The first attempt at rootcrs.sh -downgrade -auto failed because rac2 was still running:

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac2, number 2, and is terminating
CLSRSC-549: The root script cannot downgrade this node rac1 with last node operations because the Grid Infrastructure stack is up on at least one other cluster node.

Because rac1 was the node being processed with last-node downgrade operations, the GI stack on the other node had to be stopped temporarily.

On rac2 as root:

/u01/app/19.3.0/grid/bin/crsctl stop crs

9. Complete the last-node downgrade on rac1

On rac1 as root:

/u01/app/23.0.0/grid/crs/install/rootcrs.sh -downgrade -auto

This time the command can perform the required last-node operations because no other CSS daemon is active in the cluster.

10. Start the 19c Clusterware stack

After the downgrade succeeds, start rac1 explicitly from the 19c Grid home:

/u01/app/19.3.0/grid/bin/crsctl start crs -wait

Then restart the 19c stack on rac2, which was stopped for the last-node downgrade:

/u01/app/19.3.0/grid/bin/crsctl start crs -wait

11. Update Oracle Inventory on a failed node

Run the commands generated by gridSetup.sh as the grid owner.

Mark the failed 26ai Grid home as no longer being the CRS home:

$ /u01/app/23.0.0/grid/oui/bin/runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/23.0.0/grid "CLUSTER_NODES=rac1" -doNotUpdateNodeList

Then register the 19c Grid home as the CRS home again:

$ /u01/app/23.0.0/grid/oui/bin/runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/19.3.0/grid "CLUSTER_NODES=rac1" -doNotUpdateNodeList

These are not manually invented cleanup commands; they are the exact inventory commands produced by gridSetup.sh -downgrade for this failed upgrade.

12. Verify that the rollback is fully complete

Check both nodes:

/u01/app/19.3.0/grid/bin/crsctl check cluster -all

Verify their software versions:

/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac1
/u01/app/19.3.0/grid/bin/crsctl query crs softwareversion rac2

Both should report:

19.0.0.0.0

Finally:

/u01/app/19.3.0/grid/bin/crsctl query crs activeversion -f

The rollback is complete when the cluster reports:

Oracle Clusterware active version on the cluster is [19.0.0.0.0].The cluster upgrade state is [NORMAL].

Only after reaching NORMAL should another GI upgrade attempt be started.

PRVG-11960 : Set user ID bit is not set for file oradism

Problem:

While running asmca, I have got the following error:

Cause - Following nodes does not have required file ownership/permissions: Node :mk23ai-b PRVG-11960 : Set user ID bit is not set for file "/u01/app/oracle/product/23ai/dbhome_1/bin/oradism" on node "mk23ai-b".   Action - Run the Oracle home root script as the "root" user to fix the permissions.

Troubleshoot:

Check the current permissions on the file:

oracle@mk23ai-b:~$ ll /u01/app/oracle/product/23ai/dbhome_1/bin/oradism
-rwxr-x---. 1 root oinstall 1138016 Jul 11 2024 /u01/app/oracle/product/23ai/dbhome_1/bin/oradism

Solution:

The error message includes an action section that states the steps to follow. Connect to the database server as the root user and execute the root.sh script from the RDBMS home directory, since oradism mentioned in the error is located there.

root@mk23ai-b:~# /u01/app/oracle/product/23ai/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/23ai/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Check the file permissions again:

oracle@mk23ai-b:~$ ll /u01/app/oracle/product/23ai/dbhome_1/bin/oradism
-rwsr-x---. 1 root oinstall 1138016 Jul 11 2024 /u01/app/oracle/product/23ai/dbhome_1/bin/oradism

This time it has user ID bit is set.

Normally, when you run a program (an executable file), it runs with your own permissions – meaning it can only do what your user account is allowed to do. But if the setuid bit is set on a file, the program runs with the permissions of the file’s owner, regardless of who is running it.

You can continue using ASMCA this time.

CLSRSC-400: A system reboot is required to continue installing during root.sh

Problem:

While running the root.sh script, I encountered the following error:

root@mk23ai-b:~# /u01/app/23ai/grid/root.sh
Performing root user operation.
...
/u01/app/oracle/crsdata/mk23ai-b/crsconfig/roothas_2025-05-16_07-10-12PM.log
2025/05/16 19:10:17 CLSRSC-400: A system reboot is required to continue installing.
Died at /u01/app/23ai/grid/crs/install/oraafd.pm line 688.

If you think rebooting the system and rerunning the root.sh script will help, well, no… I’ve already tried :).

Troubleshoot:

Check if the secure boot is enabled:

root@mk23ai-b:~# mokutil --sb-state
SecureBoot enabled

Based on the previous output, Secure Boot is enabled. If you received a different output, such as “Secure Boot disabled,” continue your search.

Solution:

To disable Secure Boot, you cannot do it from the terminal; you need to access your computer’s firmware settings (BIOS or UEFI) and disable the Secure Boot option there.

In my case, I am using an Azure VM, and I can show you how I disabled it using the Azure console.

Note: This action requires VM downtime.

VM -> Configuration -> Security type section -> uncheck “Enable secure boot” -> Apply

Connect to the VM and rerun root.sh script:

root@mk23ai-b:~# /u01/app/23ai/grid/root.sh
...

mk23ai-b 2025/05/16 19:32:48 /u01/app/oracle/crsdata/mk23ai-b/olr/backup_20250516_193248.olr 0
2025/05/16 19:32:50 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

OPatchauto fails: CLSRSC-180: An error occurred while executing the command ‘/bin/rpm -qf /sbin/init’

Problem:

During applying ACFS patch on top of GI home, I received the following error:

Command failure output:
...
2024/03/08 19:31:03 CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init'

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Fri Mar 8 19:31:04 2024
Time taken to complete the session 3 minutes, 38 seconds

opatchauto failed with error code 42

Troubleshooting:

I attempted to manually execute the command that failed, and it returned a helpful error message:

[root@rac1 tmp]# /bin/rpm -qf /sbin/init
error: rpmdb: BDB0113 Thread/process 5003/139974823143296 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 - (-30973)
error: cannot open Packages database in /var/lib/rpm
...

Solution:

I have a solution for this type of error in another post. Let’s solve it again:

[root@rac1 tmp]# rpm  --rebuilddb

Rerun the failing command to make sure it was resolved:

[root@rac1 tmp]# /bin/rpm -qf /sbin/init
systemd-239-78.0.3.el8.x86_64

If you encountered an error during patching, you can resume opatchauto at this point:

[root@rac1 tmp]# /u01/app/19.3.0/grid/OPatch/opatchauto resume

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /tmp/36114443/36114443
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-03-08_19-29-48PM_1.log

Good luck, as always!

CRS-2549: Resource ‘ora.asmgroup’ cannot be placed on ‘rac1’ as it is not a valid candidate as per the placement policy

Problem:

After failed JDK patching on the 1st node, we tried troubleshooting and saw that ASM was not able to start:

# su - grid
$ sqlplus / as sysasm
SQL> startup nomount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ORA-39511: Start of CRS resource for instance '223' failed with error:[CRS-2549: Resource 'ora.asmgroup' cannot be placed on 'rac1' as it is not a valid candidate as per the placement policy
CRS-0223: Resource 'ora.asm' has placement error.
clsr_start_resource:260 status:223
clsrapi_start_asm:start_asmdbs status:223

Reason:

Prepatch modified RESOURCE_USE_ENABLED=0 for rac1 node:

[grid@rac1 ~]$ crsctl stat server -f

NAME=rac1
MEMORY_SIZE=63465
CPU_COUNT=8
CPU_CLOCK_RATE=2499
CPU_HYPERTHREADING=1
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=0
SERVER_LABEL=
PHYSICAL_HOSTNAME=
CSS_CRITICAL=no
CSS_CRITICAL_TOTAL=0
RESOURCE_TOTAL=0
SITE_NAME=stsfilive
STATE=ONLINE
ACTIVE_POOLS=Free
STATE_DETAILS=
ACTIVE_CSS_ROLE=hub

NAME=rac2
MEMORY_SIZE=63465
CPU_COUNT=8
CPU_CLOCK_RATE=2499
CPU_HYPERTHREADING=1
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=1
….

Solution:

Connect to the failing node and run:

[root@rac1 ~]# crsctl set resource use 1

Start ASM.

Part 2: ora.storage fails to start, ORA-01017

Problem:

One of our customers changed ASM password file by mistake and regarding other actions, we are not sure. After node restart, they encountered ora.storage startup issue on the second node.

CRS-2672: Attempting to start 'ora.storage' on 'orcl02'
ORA-01017: invalid username/password; logon denied
CRS-5055: unable to connect to an ASM instance because no ASM instance is running in the cluster
CRS-2883: Resource 'ora.storage' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
CRS-4000: Command Start failed, or completed with errors.

I have followed my blog post to recover ASM passwordfile and add CRSUSER__ASM_001. The CRS started successfully on the first node but it still was not able to start on the second.

Reason:

When we checked password for CRSUSER__ASM_001 on both nodes, we got different results:

[grid@orcl01 ~]$ crsctl get credmaint -path ASM/Self/0b5330fe4bdf6f6ebffb09beab078d6e -credtype userpass -id 0 -attr passwd -local 
zSZDts1PQx8v7gRrdmH1EjIpSBsAt
[grid@orcl02 ~]$ crsctl get credmaint -path ASM/Self/0b5330fe4bdf6f6ebffb09beab078d6e -credtype userpass -id 0 -attr passwd -local 
rHgulYGfY17Uxbb9Tbd9VF3yr2Kvr

Which is not normal and they must be the same. This was the reason CRS was not able to start on the second node, because ASM passwordfile for CRSUSER__ASM_001 had value zSZDts1PQx8v7gRrdmH1EjIpSBsAt

Solution:

Verify and fix the credentials:

If you are not able to set up root ssh passwordless connectivity, you can run the following command as grid. Note in that case you will get “credfix: could not delete crs credentials for jxrucJl3”, this is because the command was not run as root and old credentials were not deleted. But new credentials are successfully created.

[grid@orcl01 ~]$ asmcmd --nocp credverify
credverify: More than one credential in password file, please run 'credfix' to fix the credentials.
​
[grid@orcl01 ~]$ asmcmd --nocp credfix
credfix: Credentials for JXRUCJL3 not in password file, trying next credential.
op=addcrscreds wrap=/tmp/creds0.xml
credfix: Creating new credentials, no valid credentials in OCR.
credfix: New user CRSUSER__ASM_004 created.
op=credimport wrap=/tmp/creds0.xml olr=true force=true
credfix: OLR for orcl01 has been fixed if credentials were created incorrectly.
credfix: Starting SSH session on node orcl02.
credfix: OLR for orcl02 has been fixed if credentials were created incorrectly. Exiting SSH session.
op=delcrscreds crs_user=jxrucJl3
ASMCMD-8202: internal error:
credfix: could not delete crs credentials for jxrucJl3

It is recommended to setup passwordless ssh connectivity for root user and then run credfix as root to have clean configuration without old entries:

[root@rac1 ~]# asmcmd --nocp credfix
..

ora.evmd and ora.mdnsd fails to start when http_proxy is set to https://

Problem:

After setting http_proxy to https string (export http_proxy=https://test) and then stopping and starting CRS got the following error:

CRS-2883: Resource 'ora.evmd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
PRVG-2031 : Owner of file "/u01/app/19.3.0/grid/bin/CommonSetup.pm" did not match the expected value on node "rac1". [Expected = "root(0)" ; Found = "grid(3002)"]
....
PRVG-2031 : Owner of file "/u01/app/19.3.0/grid/lib/libnl19.a" did not match the expected value on node "rac1". [Expected = "root(0)" ; Found = "grid(3002)"]
CRS-4000: Command Start failed, or completed with errors.

Even after unsetting http_proxy and trying to stop CRS got the following:

[root@rac1 ~]# crsctl start crs -wait
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

[root@rac1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2679: Attempting to clean 'ora.mdnsd' on 'rac1'
CRS-2679: Attempting to clean 'ora.gpnpd' on 'rac1'
CRS-2679: Attempting to clean 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2680: Clean of 'ora.evmd' on 'rac1' failed
CRS-2680: Clean of 'ora.gpnpd' on 'rac1' failed
CRS-2680: Clean of 'ora.mdnsd' on 'rac1' failed
CRS-2799: Failed to shut down resource 'ora.evmd' on 'rac1'
CRS-2799: Failed to shut down resource 'ora.gpnpd' on 'rac1'
CRS-2799: Failed to shut down resource 'ora.mdnsd' on 'rac1'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors

So https entry in http_proxy variable caused my CRS even not being able to stop.

Solution:

The solution is simple, find processes that were started during previous attempt and kill them (be careful, not to kill anything that is not started from GI home):

[root@rac1 ~]# ps -ef|grep d.bin
root      1817     1  0 05:12 ?        00:00:01 /opt/flashgrid/bin/flashgrid_aio_srv
root      1821     1  0 05:12 ?        00:00:06 /opt/flashgrid/bin/flashgrid_target_srv
root      1824     1  0 05:12 ?        00:00:13 /opt/flashgrid/bin/flashgrid_initiator_srv
grid      1832     1  0 05:12 ?        00:00:04 /opt/flashgrid/bin/flashgrid_asm_srv
root      1845     1  0 05:12 ?        00:00:06 /opt/flashgrid/bin/flashgrid_cluster_srv
root      1879     1  0 05:12 ?        00:00:02 /opt/flashgrid/bin/flashgrid_iamback
root      1881     1  0 05:12 ?        00:00:00 /opt/flashgrid/bin/flashgrid_diskwatch
root      1884     1  0 05:12 ?        00:00:00 /opt/flashgrid/bin/flashgrid_reconstruct
root     10228 13775  0 05:43 pts/0    00:00:00 grep --color=auto d.bin
root     20305     1  2 05:16 ?        00:00:33 /u01/app/19.3.0/grid/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.US7ASCII
root     20631     1  0 05:16 ?        00:00:05 /u01/app/19.3.0/grid/bin/orarootagent.bin

[root@rac1 ~]# kill -9 20305 20631

[root@rac1 ~]# ps -ef|grep d.bin
root      1817     1  0 05:12 ?        00:00:01 /opt/flashgrid/bin/flashgrid_aio_srv
root      1821     1  0 05:12 ?        00:00:06 /opt/flashgrid/bin/flashgrid_target_srv
root      1824     1  0 05:12 ?        00:00:13 /opt/flashgrid/bin/flashgrid_initiator_srv
grid      1832     1  0 05:12 ?        00:00:04 /opt/flashgrid/bin/flashgrid_asm_srv
root      1845     1  0 05:12 ?        00:00:06 /opt/flashgrid/bin/flashgrid_cluster_srv
root      1879     1  0 05:12 ?        00:00:02 /opt/flashgrid/bin/flashgrid_iamback
root      1881     1  0 05:12 ?        00:00:00 /opt/flashgrid/bin/flashgrid_diskwatch
root      1884     1  0 05:12 ?        00:00:00 /opt/flashgrid/bin/flashgrid_reconstruct
root     10296 13775  0 05:43 pts/0    00:00:00 grep --color=auto d.bin

Make sure http_proxy is not set or instead of https there is http as a value:

[root@rac1 ~]# unset http_proxy

[root@rac1 ~]# echo $http_proxy

Or

[root@rac1 ~]# export http_proxy=http://test

Try to start CRS now:

[root@rac1 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.crf' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac1'
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-6017: Processing resource auto-start for servers: rac1
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2672: Attempting to start 'ora.chad' on 'rac1'
CRS-2672: Attempting to start 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1'
CRS-2676: Start of 'ora.chad' on 'rac1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2676: Start of 'ora.ons' on 'rac1' succeeded
CRS-6016: Resource auto-start has completed for server rac1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.

DPI-1030: unable to get or set error structure for thread local storage

Problem:

flashgrid-cluster command was showing that diskgroups were not mounted, while diskgroup were successfully mounted on all nodes:

Reason:

GI was upgraded and Flashgrid was not able to reconnect ASM.

Solution:

Restart flashgrid_asm service, please note that it does not cause any downtime and is safe to run during business hours:

 # systemctl restart flashgrid_asm.service

INS-45511: Installer has detected that an Oracle Grid Infrastructure home is marked incorrectly as configured

Problem:

After deconfiguring Oracle Restart stack using:

[root@rac1 ~]# /u01/app/19.3.0/grid/root.sh -deconfig
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac1/crsconfig/hadeconfig.log
2020/07/04 10:49:21 CLSRSC-332: CRS resources for listeners are still configured
2020/07/04 10:49:49 CLSRSC-337: Successfully deconfigured Oracle Restart stack

Tried to configure GI as clusterware stack and got the following error:

INS-45511: Installer has detected that an Oracle Grid Infrastructure home is marked incorrectly  as configured

Solution:

Remove CRS="true" accross GI home entry in /u01/app/oraInventory/ContentsXML/inventory.xml

Original:

<HOME NAME="OraGI19Home1" LOC="/u01/app/19.3.0/grid" TYPE="O" IDX="1" CRS="true"/>

After modification:

<HOME NAME="OraGI19Home1" LOC="/u01/app/19.3.0/grid" TYPE="O" IDX="1"/>

Retry configuration.