Presentation: Oracle GoldenGate Microservices Overview (with DEMO)
August 8, 2019 Leave a comment
PRVF-6402 : Core file name pattern is not same on all the nodes
August 5, 2019 Leave a comment
Problem:
Oracle 18c GI configuration prerequisite checks failed with the following error:
PRVF-6402 : Core file name pattern is not same on all the nodes. Found core filename pattern "core" on nodes "primrac1". Found core filename pattern "core.%p" on nodes "primrac2". - Cause: The core file name pattern is not same on all the nodes. - Action: Ensure that the mechanism for core file naming works consistently on all the nodes. Typically for Linux, the elements to look into are the contents of two files /proc/sys/kernel/core_pattern or /proc/sys/kernel/core_uses_pid. Refer OS vendor documentation for platforms AIX, HP-UX, and Solaris.
Comparing parameter values on both nodes:
[root@primrac1 ~]# cat /proc/sys/kernel/core_uses_pid 0 [root@primrac2 ~]# cat /proc/sys/kernel/core_uses_pid 1 [root@primrac1 ~]# sysctl -a|grep core_uses_pid kernel.core_uses_pid = 0 [root@primrac2 ~]# sysctl -a|grep core_uses_pid kernel.core_uses_pid = 1
Strange fact was that this parameter was not defined explicitly in sysctl.conf file, but still had different default values:
[root@primrac1 ~]# cat /etc/sysctl.conf |grep core_uses_pid [root@primrac2 ~]# cat /etc/sysctl.conf |grep core_uses_pid
Solution:
I’ve set parameter to 1 explicitly in sysctl.conf on both nodes:
[root@primrac1 ~]# cat /etc/sysctl.conf |grep core_uses_pid kernel.core_uses_pid=1 [root@primrac2 ~]# cat /etc/sysctl.conf |grep core_uses_pid kernel.core_uses_pid=1 [root@primrac1 ~]# sysctl -p [root@primrac2 ~]# sysctl -p [root@primrac1 ~]# sysctl -a|grep core_uses_pid kernel.core_uses_pid = 1 [root@primrac2 ~]# sysctl -a|grep core_uses_pid kernel.core_uses_pid = 1
Pressed Check Again button and GI configuration succeeded.
LGWR: Primary database is in MAXIMUM AVAILABILITY mode | ORA-16072: a minimum of one standby database destination is required
July 29, 2019 Leave a comment
Problem:
One of our customer cloned database from a DG environment to a different host and tried to open it as a standalone database. Controlfile and datafiles still considered the database in maximum availability mode.
Errors after trying to open the database:
LGWR: Primary database is in MAXIMUM AVAILABILITY mode LGWR: Destination LOG_ARCHIVE_DEST_1 is not serviced by LGWR LGWR: Minimum of 1 LGWR standby database required Thu Jul 18 18:43:14 2019 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_lgwr_39735_39805.trc: ORA-16072: a minimum of one standby database destination is required
Solution:
SQL> startup mount; SQL> alter database set standby database to maximize performance; SQL> shutdown immediate; $ srvctl start database -d orcl
UDEV rules for configuring ASM disks
July 22, 2019 Leave a comment
Problem:
During my previous installations I used the following udev rule on multipath devices:
KERNEL=="dm-[0-9]*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360050768028200a9a40000000000001c", NAME="oracleasm/asm-disk1", OWNER="oracle", GROUP="asmadmin", MODE="0660"
So to identify the exact disk I used PROGRAM option. The above script looks through `/dev/dm-*` devices and if any of them satisfy the condition, for example:
# scsci_id -gud /dev/dm-3 360050768028200a9a40000000000001c
then device name will be changed to /dev/oracleasm/asm-disk1, owner:group to grid:asmadmin and permission to 0660
But on my new servers same udev rule was not working anymore. (Of course, it needs more investigation, but our time is really valuable and never enough and if we know another solution that works and is acceptable- let’s just use it)
Solution:
I used udevadm command to identify other properties of these devices and wrote new udev rule (to see all properties, just remove grep):
# udevadm info --query=property --name /dev/mapper/asm1 | grep DM_UUID DM_UUID=mpath-360050768028200a9a40000000000001c
New udev rule looks like this:
# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
ENV{DM_UUID}=="mpath-360050768028200a9a40000000000001c", SUBSYSTEM=="block", NAME="oracleasm/asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
Trigger udev rules:
# udevadm trigger
Verify that name, owner, group and permissions are changed:
# ll /dev/oracleasm/
total 0
brw-rw---- 1 grid asmadmin 253, 3 Jul 17 17:33 asm-disk1
TNS-12518: TNS:listener could not hand off client connection | TNS-12547: TNS:lost contact
July 18, 2019 Leave a comment
Problem:
In two-node cluster, client was not able to connect to the second node, but connection to the first node was successful.
Connection from SQL developer threw error: Status: Failure - Test failed: IO Error: Got minus one from a read call, connect lapse 16ms, Authentication lapse 0ms
Connection from sqlplus using TNS string showed:
[oracle@rac02 ~]$ sqlplus "sys@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac02.example.com)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=orcl)))" as sysdba ORA-12537: TNS:connection closed
Listener.log showed:
2019-07-18T11:19:23.568231+00:00
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
Solution:
This problem can happen in other cases (entries in sqlnet.ora .. in our case it was ok) and we could think about network problem, because initially we were trying to connect from the application sever and from the SQL developer remotely. But after getting ORA-12537: TNS:connection closed error while trying to connect via sqlplus from the local server, we could only think about local non-network related problem.
The reason of this problem was that setuid bit was not set on /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle binary:
Problematic node:
[root@rac02 ~]# ll /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle -rwxr-s--x 1 oracle asmadmin 408607040 Apr 4 19:51 /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle
Healthy node:
[oracle@rac01 ~]$ ll /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle -rwsr-s--x 1 oracle asmadmin 408607040 Apr 4 19:48 /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle
We have set setuid bit on oracle binary in RDBMS home:
[root@rac02 ~]# chmod u+s /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle [root@rac02 ~]# ll /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle -rwsr-s--x 1 oracle asmadmin 408607040 Apr 4 19:51 /u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle
The problem was resolved without restarting the database instance, so clients were able to connect to the 2nd node. But because of it was staging cluster – I still restarted the database, I just made sure that database was started with the correct binary.
FlashGrid SkyCluster Now Supports Oracle Database 19c
July 15, 2019 4 Comments
FlashGrid SkyCluster Version 19.06 now has full support of GI/DB 19c, which means using FG launcher tool (https://www.flashgrid.io/skycluster-for-aws/#launch , https://www.flashgrid.io/skycluster-for-azure/#launch , https://www.flashgrid.io/skycluster-for-gcp/#launch ) , you can setup multi-node Real Application Clusters in the cloud automatically in about 2 hours.
“Oracle 19c is a long-term support release from Oracle with extended support available through 2026”
https://www.kb.flashgrid.io/release-notes/cloud-provisioningFlashGrid
PRCS-1007 : Server pool RPPCOMS already exists
July 3, 2019 Leave a comment
Problem:
While adding database using srvctl add database command got PRCS-1007 and PRCR-1086 errors:
$ srvctl add database -db RPPCOMS -oraclehome /u01/app/oracle/product/12.1.0/dbhome_1 -spfile +DATA/RPPCOMS/spfileRPPCOMS.ora -pwfile +DATA/RPPCOMS/PASSWORD/pwdrppcoms.256.1005727427 -dbname RPPCOMS -diskgroup FRA,DATA PRCS-1007 : Server pool RPPCOMS already exists PRCR-1086 : server pool ora.RPPCOMS is already registered
Solution:
Remove mentioned server pool via root user:
# crsctl delete serverpool ora.RPPCOMS
And retry adding database service.
19cGI & 12cRDBMS opatchauto: Re-link fails on target “procob”
June 24, 2019 4 Comments
Problem:
Environment: 19c GI | 12c RDBMS | RHEL 7.6
While applying Apr 2019 RU on top of the RDBMS home, opatchauto failed:
# /u01/app/oracle/product/12.2.0/dbhome_1/OPatch/opatchauto apply /u01/swtmp/29314339/ -oh /u01/app/oracle/product/12.2.0/dbhome_1 ==Following patches FAILED in apply: Patch: /u01/swtmp/29314339 Log: /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-06-24_18-51-22PM_1.log Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "procob". Re-link fails on target "proc".
We’ve tried using opatch instead of opatchauto and it succeeded:
$ /u01/app/oracle/product/12.2.0/dbhome_1/OPatch/opatch apply Patch 29314339 successfully applied. OPatch succeeded.
The problem is definitelly related to opatchauto . We’ve opened SR to Oracle and I will update this post as soon as I get the solution from them. Before that, I want to share three workarounds for this problem with you.
Workarounds:
1. The first workaround, as you have already guessed is to use opatch instead of opatchauto.
$ /u01/app/oracle/product/12.2.0/dbhome_1/OPatch/opatch apply
2. The second workaround is to edit actions.xml file under PSU (e.g /u01/swtmp/29314339/etc/config/actions.xml) and remove the following entries (see Doc ID 2056670.1):
<oracle.precomp.lang opt_req="O" version="12.2.0.1.0"> <make change_dir="%ORACLE_HOME%/precomp/lib" make_file="ins_precomp.mk" make_target="procob"/> </oracle.precomp.lang> <oracle.precomp.common opt_req="O" version="12.2.0.1.0"> <make change_dir="%ORACLE_HOME%/precomp/lib" make_file="ins_precomp.mk" make_target="proc"/> </oracle.precomp.common>
Retry opatchauto.
3. The third workaround is to backup libons.so file in GI home and then copy from RDBMS home (You should return it back, after opatchauto succeeds).
# mv /u01/app/19.3.0/grid/lib/libons.so /u01/app/19.3.0/grid/lib/libons.so_backup # cp /u01/app/oracle/product/12.2.0/dbhome_1/lib/libons.so /u01/app/19.3.0/grid/lib/libons.so
The reason I’ve used this workaround is that, I’ve found opatchauto was using libons.so file from GI home instead of RDBMS:
[WARNING]OUI-67200:Make failed to invoke " /usr/bin/make -f ins_precomp.mk proc ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1"…. '/u01/app/19.3.0/grid/lib/libons.so: undefined reference to `memcpy@GLIBC_2.14'
If we compare libons.so files between GI and RDBMS homes, we will find that they are not the same. The problem is that opatchauto uses that file from wrong location. As long as it is not easy to find a way to force opatchauto to use libons.so file from the correct location (working on this with Oracle Support), this workaournd can also be considered.
Please keep in mind, you must return GI libons.so back after opatchauto succeeds. We don’t know what happens if we have 12c libons.so file under 19c GI.
DBA Knowledge Base 