Apache Derby: “ERROR 25502: An SQL data change is not permitted for a read-only connection, user or database.”

Problem:

During testing Oracle ZDM (Zero Downtime Migration), I’ve stopped ZDM host ungracefully. After that zdmservice was not able to start:

[zdmuser@primracq ~]$ /u01/app/zdmhome/bin/zdmservice start
...
CRS_ERROR:TCC-0004: The container was not able to start.
CRS_ERROR:One or more listeners failed to start. Full details will be found in the appropriate container log fileContext [/rhp] startup failed due to previous errors
...
Start operation could not start zdmservice.
zdmservice start failed…

Troubleshooting:

Check logs under [/rhp], in my case /u01/app/zdmbase/crsdata/primracq/rhp/logs/

[zdmuser@primracq ~]$ ll /u01/app/zdmbase/crsdata/primracq/rhp/logs/
 total 3748
-rw-r--r-- 1 zdmuser zdm   277774 Dec 17 19:19 catalina.2019-12-17.log
-rw-r--r-- 1 zdmuser zdm   186117 Dec 18 18:12 catalina.2019-12-18.log
-rw-r--r-- 1 root    root   24133 Dec 19 19:12 catalina.2019-12-19.log
-rw-r--r-- 1 root    root  541694 Dec 22 20:31 catalina.2019-12-22.log
-rw-r--r-- 1 zdmuser zdm    35666 Dec 23 12:40 catalina.2019-12-23.log
-rw-r--r-- 1 zdmuser zdm  2407571 Dec 23 12:40 catalina.out
-rw-r--r-- 1 zdmuser zdm    12541 Dec 23 12:40 derby.log
-rw-r--r-- 1 zdmuser zdm   273095 Dec 23 12:40 jwc_checker_stdout_err_0.log
-rw-r--r-- 1 zdmuser zdm      155 Dec 23 12:40 JWCStartEvent.log
-rw-r--r-- 1 zdmuser zdm     5083 Dec 17 19:19 localhost.2019-12-17.log
-rw-r--r-- 1 zdmuser zdm     5083 Dec 18 18:12 localhost.2019-12-18.log
-rw-r--r-- 1 root    root    5083 Dec 19 19:12 localhost.2019-12-19.log
-rw-r--r-- 1 root    root    5083 Dec 22 20:31 localhost.2019-12-22.log
-rw-r--r-- 1 zdmuser zdm    18708 Dec 23 12:40 localhost.2019-12-23.log

catalina.out showed the following error:

Internal Exception: java.sql.SQLException: An SQL data change is not permitted for a read-only connection, user or database.
...
Caused by: ERROR 25502: An SQL data change is not permitted for a read-only connection, user or database.

Solution:

Remove db.lck file under /u01/app/zdmbase/derbyRepo:

# rm -rf /u01/app/zdmbase/derbyRepo/db.lck

And start the service again.

Change Oracle DB name using NID

Details:

Environment: RAC
Source name: orclA
Target name: orcl

1. Make sure that you have a recoverable whole database backup.

2. In Real Application Cluster we need to set cluster_database parameter to false and mount an instance on only one node:

[oracle@primrac1 ~]$ sqlplus / as sysdba
SQL> alter system set cluster_database=false scope=spfile;

[oracle@primrac1 ~]$ srvctl stop database orclA
[oracle@primrac1 ~]$ sqlplus / as sysdba
SQL> startup mount;

3. From the 1st node run nid utility. Specify username with sysdba privilege, target database name, and SETNAME parameter to yes.

[oracle@primrac1 ~]$ . oraenv
ORACLE_SID = [oracle] ? orclA1
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@primrac1 ~]$ nid TARGET=SYS DBNAME=orcl SETNAME=YES
DBNEWID: Release 19.0.0.0.0 - Production on Mon Dec 16 16:54:36 2019
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Password:
Connected to database ORCLA (DBID=3133348785)
Connected to server version 19.4.0
Control Files in database:
    +DATA/ORCLA/CONTROLFILE/current.290.1020357617
Change database name of database ORCLA to ORCL? (Y/[N]) => Y
Proceeding with operation
Changing database name from ORCLA to ORCL
     Control File +DATA/ORCLA/CONTROLFILE/current.290.1020357617 - modified
     Datafile +DATA/ORCLA/DATAFILE/system.283.102035744 - wrote new name
     Datafile +DATA/ORCLA/DATAFILE/sysaux.286.102035749 - wrote new name
     Datafile +DATA/ORCLA/DATAFILE/undotbs1.287.102035753 - wrote new name
     Datafile +DATA/ORCLA/DATAFILE/undotbs2.300.102035839 - wrote new name
     Datafile +DATA/ORCLA/DATAFILE/users.288.102035754 - wrote new name
     Datafile +DATA/ORCLA/TEMPFILE/temp.296.102035765 - wrote new name
     Control File +DATA/ORCLA/CONTROLFILE/current.290.1020357617 - wrote new name
     Instance shut down
 Database name changed to ORCL.
 Modify parameter file and generate a new password file before restarting.
 Succesfully changed database name.
 DBNEWID - Completed succesfully.

5. Change db_name parameter in the initialization parameter file:

[oracle@primrac1 dbs]$ . oraenv
ORACLE_SID = [orcl1] ? orclA1

[oracle@primrac1 dbs]$ sqlplus / as sysdba
SQL> startup nomount;
SQL> alter system set db_name=orcl scope=spfile;
SQL> alter system set cluster_database=true scope=spfile;
SQL> shut immediate;

6. Modify database name in srvctl:

[oracle@primrac1 dbs]$ srvctl modify database -db orclA -dbname orcl

7. Remove existing password file entry and create a new one:

[oracle@primrac1 dbs]$ srvctl modify database -db orclA -pwfile

[oracle@primrac1 dbs]$ orapwd  dbuniquename=orclA file='+DATA/ORCLA/PASSWORD/pwdorcl.ora'
Enter password for SYS:

8. Start the database using srvctl:

[oracle@primrac1 dbs]$ srvctl start database -db orclA

Depending on your needs you may also update instance names. If so, please make sure that you don’t have instance-specific parameters (e.g orclA1.instance_number), otherwise, it’s better to recreate the parameter file with the correct instance names.

I’ve shown you just a simple example of the database name change. In your specific case, there may be other things that depend on the database name.

Common usage examples of cluvfy utility

1. Discovering all available network interfaces and verifying the connectivity between the nodes in the cluster through those network interfaces:

$ cluvfy comp nodecon -n all -verbose

2. Check the reachability of specified nodes from a source node:

$ cluvfy comp nodereach -n all -verbose

3. Check the integrity of Oracle Cluster Registry (OCR) on all the specified nodes:

$ cluvfy comp ocr -n all -verbose

4. Verifying the integrity of the Oracle high availability services daemon on all nodes in the cluster:

$ cluvfy comp ohasd -n all -verbose

5. Comparing nodes:

$ cluvfy comp peer -n all -verbose

6. Verifying the software configuration on all nodes in the cluster for the Oracle Clusterware home directory:

$ cluvfy comp software -n all -verbose

7. Check the integrity of the cluster on all the nodes in the node list:

$ cluvfy comp clu -n all -verbose

Retrieving the Public Key for Key Pair on Linux or Mac OS

ssh-keygen command can be used on Linux or Mac OS to retrieve the public key from the private SSH key:

$ ssh-keygen -y -f MyKeyPair.pem 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChauLwkBK/vIiFY/t7uY6lzxESqZkZNvCAA3L42OH2fWzKptqGF+N32zjmLLSPFpYjoEHoHpi5e7yypTmiljtHcKUTwJTs3xclQrApCQvR+LneOi/5P5WaYl61G76osJesXiunLTa+RVr3LDR96LjPcql7JDnuh1RFhDqZ87nDfcGmXGV8iG7w3bk3R/2LuzzMYTgEVdv91S1OF1roH1baPXSV8MaYbOKhMUqV61+eP6/F5ZhT5Gk0BKX1KnQ3/gbgMqjMWRMZzYUeVjUbC52lYwrrBTQX5tHphAJtOTNJ/CpyuEuZ7ED+XYhX9Q1DNOZ47K51xbg5lsnyOBYSUqHz

-y – This option will read a private OpenSSH format file and print its public key.
-f – Specifies the filename of the key file.

Find Oracle patch description

Some patches contain several sub-patches inside. For example, when we download GI RU and unzip it, the unzipped folder contains several patch folders. To find out information about these patches you can do any of the following:

1. OPen README.html and find the section “Patch Numbers Getting Installed as Part of this Bundle Patch

2. Each patch sub-folder contains etc/config/inventory.xml, that contains description about patch:

[oracle@primrac1 29708769]$ pwd
 /home/oracle/Downloads/29708769

[oracle@primrac1 29708769]$ cat ./29834717/etc/config/inventory.xml|grep patch_description
     Database Release Update : 19.4.0.0.190716 (29834717)

[oracle@primrac1 29708769]$ cat ./29850993/etc/config/inventory.xml |grep patch_description
     OCW RELEASE UPDATE 19.4.0.0.0 (29850993)

Or we can see inventory.xml in all folders:

[oracle@primrac1 29708769]$ cat ./*/etc/config/inventory.xml |grep patch_description
     TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
     Database Release Update : 19.4.0.0.190716 (29834717)
     OCW RELEASE UPDATE 19.4.0.0.0 (29850993)
     ACFS RELEASE UPDATE 19.4.0.0.0 (29851014)

3. Another option is to use opatch lspatches or opatch query -all and query/list patch information:

[oracle@primrac1 29708769]$ pwd
 /home/oracle/Downloads/29708769

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch lspatches ./29850993|grep patch_description
 patch_description:OCW RELEASE UPDATE 19.4.0.0.0 (29850993)

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch lspatches ./29834717|grep patch_description
 patch_description:Database Release Update : 19.4.0.0.190716 (29834717)

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch query -all 29850993|grep description
  Patch description: "OCW RELEASE UPDATE 19.4.0.0.0 (29850993)"

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch query -all 29834717|grep description
  Patch description: "Database Release Update : 19.4.0.0.190716 (29834717)"

Patch 28729234 requires component(s) that are not installed in OracleHome. These not-installed components are oracle.crs:11.2.0.4.0

Problem:

Applying OCW patch 28729234 on RDBMS home failed with the following error:

[oracle@rac1 28729234]$ $ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME

Oracle Interim Patch Installer version 11.2.0.3.20
...
UtilSession failed: Patch 28729234 requires component(s) that are not installed in OracleHome. These not-installed components are oracle.crs:11.2.0.4.0

Reason:

The error message is confusing, because the real reason is in the patch itself. There is a duplication of folders and files. Custom/server directory under 28729234, also contains 28729234 directory.

Solution:

As a workaround run opatch from inner 28729234 directory:

[oracle@rac1 ~]$ cd /home/oracle/28813878/28729234/custom/server/28729234

Generate emocmrsp file:

[oracle@rac1 28729234]$ /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/ocm/bin/emocmrsp

OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates.  All rights reserved.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (ocm.rsp) was successfully created

Apply patch using the following command:

[oracle@rac1 28729234]$ $ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -ocmrf ./ocm.rsp 
...
Start OOP by Prereq process.
Launch OOP…
Oracle Interim Patch Installer version 11.2.0.3.20
Copyright (c) 2019, Oracle Corporation.  All rights reserved.
...
Applying interim patch '28729234' to OH '/u01/app/oracle/product/11.2.0/dbhome_1'
Patching component oracle.rdbms, 11.2.0.4.0…
Patch 28729234 successfully applied.

Checking that OCW is applied:

[oracle@rac1 28729234]$ $ORACLE_HOME/OPatch/opatch lspatches
28729234;OCW Patch Set Update : 11.2.0.4.190115 (28729234)
28729262;Database Patch Set Update : 11.2.0.4.190115 (28729262)

Platform IDs supported by patch are: 46. Platform ID needed is : 226

Problem:

Opatch apply failed with the following error:

Platform IDs supported by patch are: 46 Patch ( 24006111 ) is not applicable on current platform.
Platform ID needed is : 226

Reason:

You may not be familiar with Platform IDs, so the above error actually means “Downloaded patch is for Linux x86 and Linux x86-64 is required”.

Solution:

Download and apply patch for Linux x86-64.

Additional information:

Full list of Platform IDs can be found here

PlatformARU_ID
Apple Mac OS X (PowerPC)421
HP Tru64 UNIX87
HP-UX Itanium197
HP-UX Itanium (32-bit)278
HP-UX PA-RISC (32-bit)2
HP-UX PA-RISC (64-bit)59
IBM AIX on POWER Systems (32-bit)319
IBM AIX on POWER Systems (64-bit)212
IBM i on POWER Systems43
IBM S/390-based Linux (31-bit)211
IBM z/OS on System z30
IBM: Linux on POWER Systems227
IBM: Linux on System z209
Linux Itanium214
Linux x8646
Linux x86-64226
Microsoft Windows (32-bit)912
Microsoft Windows Itanium (64-bit)208
Microsoft Windows x64 (64-bit)233
Sun Solaris SPARC (32-bit)453
Sun Solaris SPARC (64-bit)23
Sun Solaris x86 (32-bit)173
Sun Solaris x86-64 (64-bit)267

ACFS Input/output error on OS kernel 3.10.0-957

Problem:

My ACFS volume has intermittent problems, when I run ls command I get ls: cannot access sm: Input/output error and user/group ownership contains question marks.

[root@primrac1 GG_HOME]# ll
ls: cannot access ma: Input/output error
ls: cannot access sm: Input/output error
ls: cannot access deploy: Input/output error
total 64
d????????? ? ? ? ? ? deploy
drwx------ 2 oracle oinstall 65536 Sep 28 21:05 lost+found
d????????? ? ? ? ? ? ma
d????????? ? ? ? ? ? sm

Reason:

Doc ID 2561145.1 mentions that kernel 3.10.0-957 changed behaviour of d_splice_alias interface which is used by ACFS driver.

My kernel is also the same:

[root@primrac1 GG_HOME]# uname -r
3.10.0-957.21.3.el7.x86_64

Solution:

Download and apply patch 29963428 on GI home.

[root@primrac1 29963428]# /u01/app/19.3.0/grid/OPatch/opatchauto apply -oh /u01/app/19.3.0/grid

JDBC 11g: SQLException(“Io exception: Connection reset”)

Problem:

Connection using 11g ojdbc was very slow and most of the time was failing with Connection reset error after 60s (default inbound connection timeout). Database alert log contained WARNING: inbound connection timed out (ORA-3136) errors.

Reason:

Oracle 11g JDBC drivers use random numbers during authentication. Those random numbers are generated by OS using /dev/random and if there is faulty/slow hardware or not too much activity on the system this generation can be slow, which causes slowness during jdbc connection.

Solution:

Instead of /dev/random indicate non-blocking /dev/urandom as java command line argument:

# java -Djava.security.egd=file:/dev/urandom -cp ojdbc8.jar:. JDBCTest "stbyrac-scan.example.com"

Simple java code to test connection to Oracle database

Assuming that client computer does not have Oracle client installed.

1. Download necessary version of ojdbc jar file from Oracle. The latest version for now is ojdbc8.jar

2. Install java development tools:

# yum install java-devel * -y

3. Create a sample java code, which:
– connects to the database
– selects 1 from dual
– disconnects

# cat JDBCTest.java
import java.sql.*;
 class JDBCTest{
     public static void main(String args[]) throws SQLException {
         Connection con = null;
         try{
             Class.forName("oracle.jdbc.driver.OracleDriver");
             String dbURL = "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=" + args[0] + ")(PORT=1521))(CONNECT_DATA=(SERVER = DEDICATED)(SERVICE_NAME=orclgg)))";
             System.out.println("jdbcurl=" + dbURL);
             String strUserID = "system";
             String strPassword = "Oracle123";
             con=DriverManager.getConnection(dbURL,strUserID,strPassword);
             System.out.println("Connected to the database.");
             Statement stmt=con.createStatement();
             System.out.println("Executing query");
             ResultSet rs=stmt.executeQuery("SELECT 1 FROM DUAL");
             while(rs.next())
                 System.out.println(rs.getInt("1"));
             con.close();
         }catch(Exception e){ System.out.println(e);}
         finally {
             con.close();
         }
 }
 }

4. Compile java code and check that *.class file was generated:

# javac JDBCTest.java

# ll  JDBCTest.*
 -rw-r--r-- 1 root root 1836 Sep 27 11:45 JDBCTest.class
 -rw-r--r-- 1 root root  925 Sep 27 11:45 JDBCTest.java

5. Run code:

# java -Djava.security.egd=file:/dev/../dev/urandom -cp ojdbc8.jar:. JDBCTest "stbyrac-scan.example.com"

jdbcurl=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=stbyrac-scan.example.com)(PORT=1521))(CONNECT_DATA=(SERVER = DEDICATED)(SERVICE_NAME=orclgg)))

Connected to the database.
Executing query…
1

Please note that -Djava.security.egd=file:/dev/../dev/urandom parameter is required to have a stable connection. I will discuss the importance of this parameter in the next post.