Copy a file over SSH without SCP

Problem:

/usr/bin/scp binary was removed from the system. Which has caused the Oracle Patching process to fail.

scp binary is provided by openssh-clients rpm, which was present on the system, but scp binary was missing.

Troubleshooting/Testing:

The workaround is to copy scp binary from a similar healthy server (keep the same version). However, transferring a file to a location where it doesn’t exist can be a bit challenging. Let’s try:

[fg@rac1 ~]$ scp /usr/bin/scp racq:/tmp/scp
bash: scp: command not found
lost connection

We got lost connection, because scp is not on racq node.

Solution:

Need to use ssh and cat commands. For most systems root user login is not enabled, so you need to place the file under /tmp and then relocate to the correct location.

In my example, I have already set up fg user equivalency, so in my case, the format will be the following:

[fg@rac1 ~]$ ssh racq cat < /usr/bin/scp ">" /tmp/scp

Connect to the remote server and copy /tmp/scp to the correct location. Reset permissions.

[root@racq tmp]# cp /tmp/scp /usr/bin/scp
[root@racq tmp]# chmod 755 /usr/bin/scp
[root@racq tmp]# chown root:root /usr/bin/scp

The transfer should be working now:

[fg@rac1 ~]$ scp /usr/bin/scp racq:/tmp/scp
scp      100%   89KB  44.0MB/s   00:00

The process worked for a binary file, so it will work for a text file too.

OPATCHAUTO-72050: System instance creation failed, cluvfy fails Verifying ‘/tmp/’ …FAILED (PRVF-7546)

Problem:

While running cluvfy we get the following error:

[grid@rac1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
...
Failures were encountered during execution of CVU verification request "stage -pre crsinst".
...
Verifying '/tmp/' ...FAILED
Cannot run program "/usr/bin/scp": error=13, Permission denied

You should not continue patching when runcluvfy fails, strongly recommended to solve all failed items and only after that run opatchauto. But if you do insist to run patching then you will get the following:

[root@rac1 33509923]# /u01/app/19.3.0/grid/OPatch/opatchauto apply /u01/app/patchinstall/33509923 -oh /u01/app/19.3.0/grid
OPatchauto session is initiated at Sun Mar 20 06:27:17 2022

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-03-20_06-27-42AM.log.
...
OPATCHAUTO-72050: System instance creation failed.
OPATCHAUTO-72050: Failed while retrieving system information.
OPATCHAUTO-72050: Please check log file for more details.

Reason:

The issue in our case was that /usr/bin/scp did not have correct permissions. It had 600 while it should have had 755. Why this happened? Don’t really know… it should not be happening.

Solution:

Set correct permissions on both nodes for scp binary using the following way:

# chmod 755 /bin/scp

Retry patching, in our case it was successful:

Oracle 12c relink resets oracle group to oinstall, while 19c sets asmadmin

Problem:

In a mixed environment when you have 19c GI, but several versions of Oracle RDBMS home (19c, 12c), initially during 12c database restore you may encounter the following error:

ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete

There are several reasons for the above error and one of them is when the $ORACLE_HOME/bin/oracle binary under rdbms home does not have group asmadmin. After identifying it, you try to change permissions to oracle:asmadmin and but after a while, you may notice that the group is reset to oinstall. It can happen after patching or after running relink manually. So let’s see test case:

[oracle@rac1 lib]$ ll /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
-rwsr-s--x 1 oracle asmadmin 323613256 Feb 19 17:00 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

Run relink:

[oracle@rac1 lib]$ make -f ins_rdbms.mk ioracle

Check permissions again:

[oracle@rac1 lib]$ ll /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
-rwsr-s--x 1 oracle oinstall 323613256 Feb 19 17:03 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

If you do the same for 19c home, group is still asmadmin.

Reason:

ins_rdbms.mk script is different in 19c and 12c, comparing ioracle section:

12c:

ioracle: preinstall $(ORACLE)
        -$(NOT_EXIST) $(ORACLE_HOME)/bin/oracle ||\
           mv -f $(ORACLE_HOME)/bin/oracle $(ORACLE_HOME)/bin/oracleO
        -mv $(ORACLE_HOME)/rdbms/lib/oracle $(ORACLE_HOME)/bin/oracle
        -chmod 6751 $(ORACLE_HOME)/bin/oracle

19c:

ioracle: preinstall $(ORACLE)
        -$(RMF) $(ORACLE_HOME)/bin/oracle
        -mv $(ORACLE_HOME)/rdbms/lib/oracle $(ORACLE_HOME)/bin/oracle
        -chmod 6751 $(ORACLE_HOME)/bin/oracle

        -(if [ ! -f $(ORACLE_HOME)/bin/crsd.bin ]; then \
            getcrshome="$(ORACLE_HOME)/srvm/admin/getcrshome" ; \
            if [ -f "$$getcrshome" ]; then \
                crshome="`$$getcrshome`"; \
                if [ -n "$$crshome" ]; then \
                    if [ $$crshome != $(ORACLE_HOME) ]; then \
                        oracle="$(ORACLE_HOME)/bin/oracle"; \
                        $$crshome/bin/setasmgidwrap oracle_binary_path=$$oracle; \
                    fi \
                fi \
            fi \
        fi\
        );

As you see ioracle section in 12c does not contain the last IF statement which sets oracle binary group to asmadmin (using setasmgidwrap)

Solution:

Unfortunately, I cannot recommend adding that if statement in 12c ins_rdbms.mk although I did it in my test environment and works as expected.
After running make -f ins_rdbms.mk ioracle for 12c the group is set to asmadmin.

It is better to ask Oracle about this change and do only after you get an approval from them.

The only non-harmful recommendation from my side would be to have permission change script and run after every patch or relink:

# chown oracle:asmadmin  /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
​# chmod 6751  /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
# ll   /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

If you still decide to add IF statement as I did, please make sure there is TAB instead of spaces before IF clause. Otherwise, you will get the following error and you can easily understand which line should be corrected.

[oracle@rac1 lib]$ make -f ins_rdbms.mk ioracle
ins_rdbms.mk:120: *** missing separator (did you mean TAB instead of 8 spaces?).  Stop.

Find Oracle patch description

Some patches contain several sub-patches inside. For example, when we download GI RU and unzip it, the unzipped folder contains several patch folders. To find out information about these patches you can do any of the following:

1. OPen README.html and find the section “Patch Numbers Getting Installed as Part of this Bundle Patch

2. Each patch sub-folder contains etc/config/inventory.xml, that contains description about patch:

[oracle@primrac1 29708769]$ pwd
 /home/oracle/Downloads/29708769

[oracle@primrac1 29708769]$ cat ./29834717/etc/config/inventory.xml|grep patch_description
     Database Release Update : 19.4.0.0.190716 (29834717)

[oracle@primrac1 29708769]$ cat ./29850993/etc/config/inventory.xml |grep patch_description
     OCW RELEASE UPDATE 19.4.0.0.0 (29850993)

Or we can see inventory.xml in all folders:

[oracle@primrac1 29708769]$ cat ./*/etc/config/inventory.xml |grep patch_description
     TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
     Database Release Update : 19.4.0.0.190716 (29834717)
     OCW RELEASE UPDATE 19.4.0.0.0 (29850993)
     ACFS RELEASE UPDATE 19.4.0.0.0 (29851014)

3. Another option is to use opatch lspatches or opatch query -all and query/list patch information:

[oracle@primrac1 29708769]$ pwd
 /home/oracle/Downloads/29708769

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch lspatches ./29850993|grep patch_description
 patch_description:OCW RELEASE UPDATE 19.4.0.0.0 (29850993)

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch lspatches ./29834717|grep patch_description
 patch_description:Database Release Update : 19.4.0.0.190716 (29834717)

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch query -all 29850993|grep description
  Patch description: "OCW RELEASE UPDATE 19.4.0.0.0 (29850993)"

[oracle@primrac1 29708769]$ $ORACLE_HOME/OPatch/opatch query -all 29834717|grep description
  Patch description: "Database Release Update : 19.4.0.0.190716 (29834717)"

Platform IDs supported by patch are: 46. Platform ID needed is : 226

Problem:

Opatch apply failed with the following error:

Platform IDs supported by patch are: 46 Patch ( 24006111 ) is not applicable on current platform.
Platform ID needed is : 226

Reason:

You may not be familiar with Platform IDs, so the above error actually means “Downloaded patch is for Linux x86 and Linux x86-64 is required”.

Solution:

Download and apply patch for Linux x86-64.

Additional information:

Full list of Platform IDs can be found here

PlatformARU_ID
Apple Mac OS X (PowerPC)421
HP Tru64 UNIX87
HP-UX Itanium197
HP-UX Itanium (32-bit)278
HP-UX PA-RISC (32-bit)2
HP-UX PA-RISC (64-bit)59
IBM AIX on POWER Systems (32-bit)319
IBM AIX on POWER Systems (64-bit)212
IBM i on POWER Systems43
IBM S/390-based Linux (31-bit)211
IBM z/OS on System z30
IBM: Linux on POWER Systems227
IBM: Linux on System z209
Linux Itanium214
Linux x8646
Linux x86-64226
Microsoft Windows (32-bit)912
Microsoft Windows Itanium (64-bit)208
Microsoft Windows x64 (64-bit)233
Sun Solaris SPARC (32-bit)453
Sun Solaris SPARC (64-bit)23
Sun Solaris x86 (32-bit)173
Sun Solaris x86-64 (64-bit)267

Patch Planner to check and request conflict patches

Problem:

Recently, I was applying p29963428_194000ACFSRU_Linux-x86-64.zip on top of 19.4 GI home and got the following error:

==Following patches FAILED in analysis for apply:
 Patch: /u01/swtmp/29963428/29963428
 Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-08-07_10-07-56AM_1.log
 Reason: Failed during Analysis: CheckConflictAgainstOracleHome Failed, [ Prerequisite Status: FAILED, Prerequisite output: 
 Summary of Conflict Analysis:
 There are no patches that can be applied now.
 Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches : 
 29851014, 29963428
 Conflicts/Supersets for each patch are:
 Patch : 29963428
 Bug Conflict with 29851014 Conflicting bugs are: 29039918, 27494830, 29338628, 29031452, 29264772, 29760083, 28855761 ... 
 After fixing the cause of failure Run opatchauto resume

Solution:

Oracle MOS note ID 1317012.1 describes steps how to check such conflicts and request conflict/merged patches in previous:

1. Run lsinventory from the target home:

[grid@rac1 ~]$ /u01/app/19.3.0/grid/OPatch/opatch lsinventory > GI_lsinventory.txt

2. Logon to support.oracle.com -> Click the “Patch and Updates” tab -> Enter the patch number you want to apply:

2. Click Analyze with OPatch…

3. Attach GI_lsinventory.txt file created in the first step and click “Analyze for Conflict”:

4. Wait for a while and you will see the result. According to it, patch 29963428 conflicts with my current patches:

From the same screen I can “Request Patch”.

5. After clicking “Request Patch” button I got the following error:

Click “See Details”:

The message actually means that fix for the same bug is already included in currently installed 19.4.0.0.190716ACFSRU.

So I don’t have to apply 29963428 patch. I wanted to share the steps with you , because the mentioned tool is really useful.

Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases

Document is self-explanatory: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2118136.2