PRCD-1229 : An attempt to access configuration of database orcl was rejected because its version 12.2.0.1.0 differs from the program version 19.0.0.0.0

August 21, 2020 Leave a comment

Problem:

After a manual database upgrade from 12c to 19c, I was not able to start database instance using srvctl. 

[oracle@rac1 ~]$ srvctl start instance -db orcl -n rac1

PRCD-1027 : Failed to retrieve database orcl
PRCD-1229 : An attempt to access configuration of database orcl was rejected because its version 12.2.0.1.0 differs from the program version 19.0.0.0.0. Instead run the program from /u01/app/oracle/product/12.2.0/dbhome_1.

Solution:

Use srvctl upgrade from new home:

$ /u01/app/oracle/product/19.3.0/dbhome_1/bin/srvctl upgrade database -d orcl -oraclehome /u01/app/oracle/product/19.3.0/dbhome_1

Try to start again:

$ srvctl start instance -db orcl -n rac1
Advertisement

Filed under 12c, 19c, ASM Tagged with , , , ,

Identify whether database is RAC or SI using crsctl

July 21, 2020 Leave a comment

Environment:

GI – configured as a cluster
orclsingle – configured as a Single Instance
orclrac – configured as RAC
orclone – configured as RACOneNode

Q1: How to identify whether database is RAC or SI?

A1: crsctl stat res shows parameter CLUSTER_DATABASE. In case of RAC or RACOneNode the value is TRUE otherwise FALSE

# crsctl stat res ora.orclrac.db -p|grep CLUSTER_DATABASE
CLUSTER_DATABASE=true

# crsctl stat res ora.orclone.db -p|grep CLUSTER_DATABASE
CLUSTER_DATABASE=true

# crsctl stat res ora.orclsingle.db -p|grep CLUSTER_DATABASE
CLUSTER_DATABASE=false

Please note that the same can be identified from sqlplus by selecting CLUSTER_DATABASE initialization parameter.

Q2: How to identify whether my database is RAC or RACOneNode?

A2: srvctl config database shows Type, that can have the following values SINGLE, RAC, or RACOneNode.

$ srvctl config database -db orclrac|grep Type
Type: RAC

$ srvctl config database -db orclone|grep Type
Type: RACOneNode

$ srvctl config database -db orclsingle|grep Type
Type: SINGLE

Filed under 19c, RAC Tagged with , , , , , ,

ORA-12578: TNS:wallet open failed while starting the database

March 7, 2019 Leave a comment

Problem:

ORA-12578: TNS:wallet open failed encountered while starting the database using SRVCTL. Auto-login wallet doesn’t open during database startup when started using SRVCTL .

$ srvctl start database -db orcl
PRCD-1332 : failed to start database orcl
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-5017: The resource action “ora.orcl.db start” encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file ‘+DATA/ORCL/PARAMETERFILE/spfile.291.1001107969’
ORA-17503: ksfdopn:10 Failed to open file +DATA/ORCL/PARAMETERFILE/spfile.291.1001107969
ORA-12578: TNS:wallet open failed

There are two solutions:

Solution1:

Start the database in mount mode using sqlplus:

SQL> startup mount;

Open the wallet:

SQL> alter system set encryption wallet open identified by “wallet_password”;

Open the database:

SQL> alter database open;

Solution2:

Check the value of TNS_ADMIN value by executing:

$ srvctl getenv database -d orcl -t “TNS_ADMIN”
orcl:
PRKF-1128 : Environment variable TNS_ADMIN is not defined.

Set correct TNS_ADMIN value by executing:

$ srvctl setenv database -d orcl -T “TNS_ADMIN=/u01/app/oracle/product/18.3.0/dbhome_1/network/admin”

Change the following parameter in sqlnet.ora file:

From:

SQLNET.WALLET_OVERRIDE = TRUE

To:

SQLNET.WALLET_OVERRIDE = FALSE

Start the database:

$ srvctl start database -db orcl

Filed under Real World Scenarios Tagged with , ,

SRVCTL: CRS-2678, CRS-0267, CRS-5802: Unable to start the agent process

September 6, 2018 2 Comments

We had the following problem with some customer:

srvctl start database -db dbname was failing on one of the cluster nodes with the following error:

[oracle@node1 ~]$ srvctl start database -db dbname
PRCR-1079 : Failed to start resource ora.dbname.db
CRS-2674: Start of ‘ora.dbname.db’ on ‘rac1’ failed
CRS-2678: ‘ora.dbname.db’ on ‘rac1’ has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process

But during that time we were able to startup database using sqlplus:

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL> startup

ORACLE instance started.

Total System Global Area 1577058304 bytes
Fixed Size 8621136 bytes
Variable Size 805307312 bytes
Database Buffers 754974720 bytes
Redo Buffers 8155136 bytes
Database mounted.
Database opened.

It was strange and took a lot of time for me to troubleshoot this issue.

I tried many things:
* removed srvctl config using srvctl remove database -db orcl
* readded it again srvctl add database -db orcl
* readded instances
* also tried to restart crs and even the servers
but with no luck.

Then I found the following documentation Doc ID 1957360.1 on Oracle site and tried to reproduce the same problem on my lab servers and I did it.

I tried to change the ownership for the file on my test cluster on only one node:

[root@rac1 ~]# ll /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracleOUT.trc
-rw-r–r– 1 oracle oinstall 1085 Sep 5 20:17 /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracleOUT.trc
[root@rac1 ~]# ll /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracle.pid
-rw-r–r– 1 oracle oinstall 6 Sep 5 20:17 /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracle.pid
[root@rac1 ~]# chown root:root /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracle.pid
[root@rac1 ~]# chown root:root /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracleOUT.trc

I tried to startup instance using sqlplus and it was successful:

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL> startup

ORACLE instance started
Database mounted.
Database opened.

Stopped the database and tried with srvctl :

After a long wait it failed:

[oracle@rac1 ~]$ srvctl start database -db orcl
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-2674: Start of ‘ora.orcl.db’ on ‘rac1’ failed
CRS-2678: ‘ora.orcl.db’ on ‘rac1’ has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process

I also checked customer logs and found that files crsd_oraagent_oracle.pid, crsd_oraagent_oracleOUT.trc were not updated for a long time, they were older than other files.

So to solve such problem you need to assign correct owner, group and access permission for the above two files and you will be able to start database using srvctl.

[root@rac1 ~]# chown oracle:oinstall /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracle.pid
[root@rac1 ~]# chown oracle:oinstall /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracleOUT.trc
[root@rac1 ~]# chown 644 /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracle.pid
[root@rac1 ~]# chown 644 /u01/app/grid/crsdata/rac1/output/crsd_oraagent_oracleOUT.trc

You may never have such errors but if you have you know how to solve.

Filed under 12c, DB Tagged with , , , , , ,