opatch apply failed: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)

Problem:

In the two-node cluster. I applied patch 29342099 on the 1st node successfully. But the patch failed on the 2nd node:

[oracle@stbyrac2 29342099]$ $ORACLE_HOME/OPatch/opatch apply
...
Applying interim patch '29342099' to OH '/u01/app/oracle/product/19.3.0/dbhome_1'
Patching component oracle.rdbms, 19.0.0.0.0…
Patching component oracle.rdbms.rman, 19.0.0.0.0…
Patching component oracle.rdbms.rsf, 19.0.0.0.0…
ApplySession failed in system modification phase… 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: 
java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'

The patch failed in a bad phase, after failure while I was trying to check inventory on that node I was getting:

[oracle@stbyrac2 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail
...
Inventory load failed… OPatch cannot load inventory for the given Oracle Home.
LsInventorySession failed: Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/29342099 is corrupted.
java.lang.RuntimeException: No Patch exists,Please check.

Inventory knew about patch 29342099 but $ORACLE_HOME/inventory/oneoffs/29342099 did not exist.

I decided to copy folder from 1st node (where patch was successful):

[oracle@stbyrac1 ~]$ scp -r $ORACLE_HOME/inventory/oneoffs/29342099 stbyrac2:$ORACLE_HOME/inventory/oneoffs/

After that listing inventory was successful. And patch seemed to be part of the inventory. I tried to rollback it, which failed with the same permission error `/u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)`

Solution:

Check permissions on oui-patch.xml file on both nodes:

[oracle@primrac1 ~]$ ls -la /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-rw---- 1 grid oinstall 174 Oct 9 20:46 /u01/app/oraInventory/ContentsXML/oui-patch.xml
[oracle@primrac1 ~]$ ssh primrac2 ls -la /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-r--r-- 1 grid oinstall 174 Sep 29 19:38 /u01/app/oraInventory/ContentsXML/oui-patch.xml

They are different. Modify permission of that file to 660:

[root@stbyrac2 ContentsXML]# chmod 660 oui-patch.xml 

Rollback and reapply patch:

[oracle@stbyrac2 29342099]$ $ORACLE_HOME/OPatch/opatch apply
...
Applying interim patch '29342099' to OH '/u01/app/oracle/product/19.3.0/dbhome_1'
Patching component oracle.rdbms, 19.0.0.0.0…
Patching component oracle.rdbms.rman, 19.0.0.0.0…
Patching component oracle.rdbms.rsf, 19.0.0.0.0…
Patch 29342099 successfully applied.
Log file location: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatch/opatch2020-01-27_13-55-44PM_1.log
OPatch succeeded.