root.sh fails CLSRSC-331: Failure initializing entries in file ‘/etc/oracle/scls_scr/racdb1’

After failed installation of 12gR2 Oracle Cluster Memeber, during the second attempt, I received the following error while running the root.sh script:

[root@racdb1 ~]# /u01/app/12.2.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/racdb1/crsconfig/rootcrs_racdb1_2018-05-30_01-00-14PM.log
2018/05/30 13:00:17 CLSRSC-594: Executing installation step 1 of 19: ‘SetupTFA’.
2018/05/30 13:00:17 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2018/05/30 13:00:52 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2018/05/30 13:00:52 CLSRSC-594: Executing installation step 2 of 19: ‘ValidateEnv’.
2018/05/30 13:01:00 CLSRSC-363: User ignored prerequisites during installation
2018/05/30 13:01:00 CLSRSC-594: Executing installation step 3 of 19: ‘CheckFirstNode’.
2018/05/30 13:01:03 CLSRSC-594: Executing installation step 4 of 19: ‘GenSiteGUIDs’.
2018/05/30 13:01:04 CLSRSC-594: Executing installation step 5 of 19: ‘SaveParamFile’.
2018/05/30 13:01:05 CLSRSC-594: Executing installation step 6 of 19: ‘SetupOSD’.
2018/05/30 13:01:24 CLSRSC-594: Executing installation step 7 of 19: ‘CheckCRSConfig’.
2018/05/30 13:01:24 CLSRSC-594: Executing installation step 8 of 19: ‘SetupLocalGPNP’.
2018/05/30 13:01:26 CLSRSC-594: Executing installation step 9 of 19: ‘ConfigOLR’.
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Create failed, or completed with errors.
2018/05/30 13:01:27 CLSRSC-331: Failure initializing entries in file ‘/etc/oracle/scls_scr/racdb1’
The command ‘/u01/app/12.2.0/grid/perl/bin/perl -I/u01/app/12.2.0/grid/perl/lib -I/u01/app/12.2.0/grid/crs/install /u01/app/12.2.0/grid/crs/install/rootcrs.pl ‘ execution failed

ckptGridHA_racdb1.xml is a check point file. It contains information about the node name, ocr, voting disk location, GRID_HOME, ORCALE_HOME, private interconnect, public and vip IP addresses… and is located at /u01/app/grid/crsdata/racdb1/crsconfig/

Remove the mentioned file:

# rm -rf /u01/app/grid/crsdata/racdb1/crsconfig/ckptGridHA_racdb1.xml

And rerun root.sh

[root@racdb1 ~]# /u01/app/12.2.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/racdb1/crsconfig/rootcrs_racdb1_2018-05-30_01-26-56PM.log
2018/05/30 13:26:58 CLSRSC-594: Executing installation step 1 of 19: ‘SetupTFA’.
2018/05/30 13:26:59 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2018/05/30 13:26:59 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2018/05/30 13:26:59 CLSRSC-594: Executing installation step 2 of 19: ‘ValidateEnv’.
2018/05/30 13:27:01 CLSRSC-363: User ignored prerequisites during installation
2018/05/30 13:27:01 CLSRSC-594: Executing installation step 3 of 19: ‘CheckFirstNode’.
2018/05/30 13:27:03 CLSRSC-594: Executing installation step 4 of 19: ‘GenSiteGUIDs’.
2018/05/30 13:27:03 CLSRSC-594: Executing installation step 5 of 19: ‘SaveParamFile’.
2018/05/30 13:27:07 CLSRSC-594: Executing installation step 6 of 19: ‘SetupOSD’.
2018/05/30 13:27:25 CLSRSC-594: Executing installation step 7 of 19: ‘CheckCRSConfig’.
2018/05/30 13:27:25 CLSRSC-594: Executing installation step 8 of 19: ‘SetupLocalGPNP’.
2018/05/30 13:27:46 CLSRSC-594: Executing installation step 9 of 19: ‘ConfigOLR’.
2018/05/30 13:27:53 CLSRSC-594: Executing installation step 10 of 19: ‘ConfigCHMOS’.
2018/05/30 13:27:53 CLSRSC-594: Executing installation step 11 of 19: ‘CreateOHASD’.
2018/05/30 13:27:57 CLSRSC-594: Executing installation step 12 of 19: ‘ConfigOHASD’.
2018/05/30 13:28:12 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.service’
2018/05/30 13:29:09 CLSRSC-594: Executing installation step 13 of 19: ‘InstallAFD’.
2018/05/30 13:29:12 CLSRSC-594: Executing installation step 14 of 19: ‘InstallACFS’.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘racdb1’
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘racdb1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/05/30 13:30:07 CLSRSC-594: Executing installation step 15 of 19: ‘InstallKA’.
2018/05/30 13:30:10 CLSRSC-594: Executing installation step 16 of 19: ‘InitConfig’.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘racdb1’
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘racdb1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start ‘ora.evmd’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘racdb1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘racdb1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘racdb1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.gipcd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘racdb1’
CRS-2676: Start of ‘ora.diskmon’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘racdb1’ succeeded
2018/05/30 13:30:53 CLSRSC-482: Running command: ‘/u01/app/12.2.0/grid/bin/ocrconfig -upgrade grid oinstall’
CRS-2672: Attempting to start ‘ora.crf’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.storage’ on ‘racdb1’
CRS-2676: Start of ‘ora.crf’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.storage’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘racdb1’
CRS-2676: Start of ‘ora.crsd’ on ‘racdb1’ succeeded
Now formatting voting disk: +GRID.
CRS-4256: Updating the profile
Successful addition of voting disk bfc6d4b89dd54fd6bf4f4ca43552da69.
Successfully replaced voting disk group with +GRID.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE bfc6d4b89dd54fd6bf4f4ca43552da69 (+GRID/RACDB1/VOTINGFILE/vfile.258.977491875) [GRID]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘racdb1’
CRS-2677: Stop of ‘ora.crsd’ on ‘racdb1’ succeeded
CRS-2673: Attempting to stop ‘ora.crf’ on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.storage’ on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘racdb1’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘racdb1’
CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘racdb1’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘racdb1’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘racdb1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘racdb1’ succeeded
CRS-2677: Stop of ‘ora.gpnpd’ on ‘racdb1’ succeeded
CRS-2677: Stop of ‘ora.storage’ on ‘racdb1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘racdb1’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘racdb1’ succeeded
CRS-2677: Stop of ‘ora.cssd’ on ‘racdb1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘racdb1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘racdb1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘racdb1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
2018/05/30 13:31:49 CLSRSC-594: Executing installation step 17 of 19: ‘StartCluster’.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.evmd’ on ‘racdb1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘racdb1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘racdb1’
CRS-2676: Start of ‘ora.gipcd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘racdb1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘racdb1’
CRS-2676: Start of ‘ora.diskmon’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘racdb1’
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘racdb1’
CRS-2676: Start of ‘ora.ctssd’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.crf’ on ‘racdb1’
CRS-2676: Start of ‘ora.crf’ on ‘racdb1’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘racdb1’
CRS-2676: Start of ‘ora.crsd’ on ‘racdb1’ succeeded
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘racdb1’ succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: racdb1
CRS-6016: Resource auto-start has completed for server racdb1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2018/05/30 13:33:39 CLSRSC-343: Successfully started Oracle Clusterware stack
2018/05/30 13:33:39 CLSRSC-594: Executing installation step 18 of 19: ‘ConfigNode’.
2018/05/30 13:35:22 CLSRSC-594: Executing installation step 19 of 19: ‘PostConfig’.
2018/05/30 13:35:45 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded

RAC: root.sh | CRS-2672: Attempting to start ‘ora.storage’ | ORA-01017: invalid username/password

I was configuring clusterware on node1 and got the following error:

CRS-2672: Attempting to start ‘ora.storage’ on ‘node1’
ORA-01017: invalid username/password; logon denied
CRS-5017: The resource action “ora.storage start” encountered the following error:
Storage agent start action aborted. For details refer to “(:CLSN00107:)” in “/u01/app/oracle/diag/crs/node1/crs/trace/ohasd_orarootagent_root.trc”.
CRS-2883: Resource ‘ora.storage’ failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2016/09/27 05:41:01 CLSRSC-117: Failed to start Oracle Clusterware stack

Died at /u01/app/12.1.0.2/grid/crs/install/crsinstall.pm line 930.
The command ‘/u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/rootcrs.pl ‘ exe ution failed

 

/u01/app/oracle/diag/crs/node1/crs/trace/ohasd_orarootagent_root.trc file says:

2016-09-27 05:40:56.787330*:kgfn.c@6018: kgfnConnect2Int: sysasm=0 envflags=0x10 srvrflags=0x3 unam=NULL password is NULL pstr=_ocr
2016-09-27 05:40:56.787330*:kgfn.c@6194: kgfnConnect2Int: cstr=(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/12.1.0.2/grid/bin/oracle)(ARGV0=oracle+ASM1_ocr)(ENVS=’ORACLE_HOME=/u01/app/12.1.0.2/grid,ORACLE_SID=+ASM1′)(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))’)(PRIVS=(USER=root)(GROUP=root)))(enable=setuser))
2016-09-27 05:40:57.273302 : AGENT:2583111424: {0:9:3} {0:9:3} Created alert : (:CRSAGF00113:) : Aborting the command: start for resource: ora.storage 1 1

 

So why user root???

See, when I connect using root I got ORA-01017

[root@node1 ~]# . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@node1 ~]# sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 27 05:59:01 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-01017: invalid username/password; logon denied

If I connect through Oracle it is OK:

su – oracle

[oracle@node1 ~]$ . oraenv
ORACLE_SID = [LBTCI1] ? +ASM1

[oracle@node1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 27 05:59:45 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>

 

Look the connection string again there is “PROGRAM=/u01/app/12.1.0.2/grid/bin/oracle”, so let’s check file permissions.

[oracle@node1 ~]$ ll /u01/app/12.1.0.2/grid/bin/oracle
-rwsr-s–x 1 root root 295054213 Sep 27 05:26 /u01/app/12.1.0.2/grid/bin/oracle

It must be oracle:oinstall  not root:root 

chown oracle:oinstall /u01/app/12.1.0.2/grid/bin/oracle
chmod 6751 /u01/app/12.1.0.2/grid/bin/oracle

 

deconfigure(rootcrs.pl -deconfig  -verbose) crs and reconfigure(run root.sh) it again.