Install TFA v19.2.1 as Daemon in cluster

Installing TFA as root is preferable, because you will have enhanced capabilities. Such as:

  • Automatic diagnostic collections
  • Collections from remote hosts
  • Collecting of files which are not readable by Oracle software owner such as /var/log/messages or clusterware daemon logs

Download the newest Oracle Trace File Analyzer zip from Doc ID 1513912.1 ( for simplicity you can use wget ):

# wget --http-user=<Your OTN username> "https://updates.oracle.com/Orion/Services/download/TFA-LINUX_v19.2.1.zip?aru=22891540&patch_file=TFA-LINUX_v19.2.1.zip" -O TFA-LINUX_v19.2.1.zip --ask-password
Password: <Your OTN user password>

Setup passwordless ssh user equivalency for root user to all nodes:

[root@rac1 install]# ssh-keygen -t rsa <press Enter on each question>
[root@rac2 install]# ssh-keygen -t rsa <press Enter on each question> 

Copy content of /root/.ssh/id_rsa.pub from rac1 and paste into /root/.ssh/authorized_keys on rac2. Repeat the same for rac2.

Please make sure that the parameter PermitRootLogin yes exists in /etc/ssh/sshd_config file.

Check ssh equivalency:

[root@rac1 install]# ssh rac2 date
 Tue Aug  6 10:37:07 UTC 2019

[root@rac2 .ssh]# ssh rac1 date
 Tue Aug  6 10:37:34 UTC 2019

Unzip file and run the installTFA command ( bolded texts are answers to TFA installation questions, other text is only installation log):

[root@rac1 install]# unzip TFA-LINUX_v19.2.1.zip
[root@rac1 install]# ll
 total 518220
 -rwxr-xr-x 1 root root 265898937 Apr 25 18:44 installTFA-LINUX
 -rw-r--r-- 1 root root      1460 Apr 25 18:09 README.txt
 -rw-r--r-- 1 root root 264751391 Apr 25 19:18 TFA-LINUX_v19.2.1.zip

[root@rac1 install]# ./installTFA-LINUX 

TFA Installation Log will be written to File : /tmp/tfa_install_27153_2019_08_06-10_40_07.log
Starting TFA installation
TFA Version: 192100 Build Date: 201904251105
Enter a location for installing TFA (/tfa will be appended if not supplied) [/sw/install/tfa]:
/u01/app/18.3.0/grid/tfa
Running Auto Setup for TFA as user root…
Would you like to do a [L]ocal only or [C]lusterwide installation ? [L|l|C|c] [C] : C
The following installation requires temporary use of SSH.
If SSH is not configured already then we will remove SSH when complete.
Do you wish to Continue ? [Y|y|N|n] [Y] Y
Installing TFA now…
Discovering Nodes and Oracle resources
Starting Discovery…
Getting list of nodes in cluster . . . . .
List of nodes in cluster:
rac1
rac2
Checking ssh user equivalency settings on all nodes in cluster
Node rac2 is configured for ssh user equivalency for root user
CRS_HOME=/u01/app/18.3.0/grid
Searching for running databases…
orcl 
Searching out ORACLE_HOME for selected databases…
Getting Oracle Inventory…
ORACLE INVENTORY: /u01/app/oraInventory
Discovery Complete…
TFA Will be Installed on the Following Nodes:
++++++++++++++++++++++++++++++++++++++++++++
Install Nodes
rac1
rac2
Do you wish to make changes to the Node List ? [Y/y/N/n] [N] N
TFA will scan the following Directories
++++++++++++++++++++++++++++++++++++++++++++
.-------------------------------------------------------------------------.
|                                   rac1                                  |
+--------------------------------------------------------------+----------+
| Trace Directory                                              | Resource |
+--------------------------------------------------------------+----------+
| /u01/app/18.3.0/grid/cfgtoollogs                             | CFGTOOLS |
| /u01/app/18.3.0/grid/crs/log                                 | CRS      |
| /u01/app/18.3.0/grid/css/log                                 | CRS      |
| /u01/app/18.3.0/grid/evm/admin/logger                        | CRS      |
| /u01/app/18.3.0/grid/evm/log                                 | CRS      |
| /u01/app/18.3.0/grid/install                                 | INSTALL  |
| /u01/app/18.3.0/grid/inventory/ContentsXML                   | INSTALL  |
| /u01/app/18.3.0/grid/log                                     | CRS      |
| /u01/app/18.3.0/grid/network/log                             | CRS      |
| /u01/app/18.3.0/grid/opmn/logs                               | CRS      |
| /u01/app/18.3.0/grid/rdbms/log                               | ASM      |
| /u01/app/18.3.0/grid/srvm/log                                | CRS      |
| /u01/app/grid/cfgtoollogs                                    | CFGTOOLS |
| /u01/app/grid/crsdata/rac1/acfs                              | ACFS     |
| /u01/app/grid/crsdata/rac1/afd                               | ASM      |
| /u01/app/grid/crsdata/rac1/chad                              | CRS      |
| /u01/app/grid/crsdata/rac1/core                              | CRS      |
| /u01/app/grid/crsdata/rac1/crsconfig                         | CRS      |
| /u01/app/grid/crsdata/rac1/crsdiag                           | CRS      |
| /u01/app/grid/crsdata/rac1/cvu                               | CRS      |
| /u01/app/grid/crsdata/rac1/evm                               | CRS      |
| /u01/app/grid/crsdata/rac1/output                            | CRS      |
| /u01/app/grid/crsdata/rac1/trace                             | CRS      |
| /u01/app/grid/diag/apx/+apx                                  | ASMPROXY | 
| /u01/app/grid/diag/apx/+apx/+APX1/cdump                      | ASMPROXY |
| /u01/app/grid/diag/apx/+apx/+APX1/trace                      | ASMPROXY |
| /u01/app/grid/diag/asm/+asm/+ASM1/cdump                      | ASM      |
| /u01/app/grid/diag/asm/+asm/+ASM1/trace                      | ASM      |
| /u01/app/grid/diag/asmtool/user_grid/host_2173698417_110/cdu | ASMTOOL  |
| /u01/app/grid/diag/asmtool/user_grid/host_2173698417_110/tra | ASM      |
| /u01/app/grid/diag/clients/user_grid/host_2173698417_110/cdu | DBCLIENT |
| /u01/app/grid/diag/clients/user_grid/host_2173698417_110/tra | DBCLIENT |
| /u01/app/grid/diag/crs/rac1/crs/cdump                        | CRS      |
| /u01/app/grid/diag/crs/rac1/crs/trace                        | CRS      |
| /u01/app/grid/diag/rdbms/_mgmtdb/-MGMTDB/cdump               | RDBMS    |
| /u01/app/grid/diag/rdbms/_mgmtdb/-MGMTDB/trace               | RDBMS    |
| /u01/app/grid/diag/tnslsnr/rac1/listener/cdump               | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener/trace               | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener_scan1/cdump         | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener_scan1/trace         | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener_scan2/cdump         | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener_scan2/trace         | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener_scan3/cdump         | TNS      |
| /u01/app/grid/diag/tnslsnr/rac1/listener_scan3/trace         | TNS      |
| /u01/app/oraInventory/ContentsXML                            | INSTALL  |
| /u01/app/oraInventory/logs                                   | INSTALL  |
| /u01/app/oracle/cfgtoollogs                                  | CFGTOOLS |
| /u01/app/oracle/diag/clients/user_oracle/host_2173698417_110 | DBCLIENT |
| /u01/app/oracle/diag/clients/user_oracle/host_2173698417_110 | DBCLIENT |
| /u01/app/oracle/diag/rdbms/orcl/orcl1/cdump                  | RDBMS    |
| /u01/app/oracle/diag/rdbms/orcl/orcl1/trace                  | RDBMS    |
| /u01/app/oracle/diag/tnslsnr                                 | TNS      |
| /u01/app/oracle/product/18.3.0/dbhome_1/cfgtoollogs          | CFGTOOLS |
| /u01/app/oracle/product/18.3.0/dbhome_1/install              | INSTALL  |
| com                                                          | ASM      |
'--------------------------------------------------------------+----------'
Installing TFA on rac1:
HOST: rac1    TFA_HOME: /u01/app/18.3.0/grid/tfa/rac1/tfa_home
Installing TFA on rac2:
HOST: rac2    TFA_HOME: /u01/app/18.3.0/grid/tfa/rac2/tfa_home
.-------------------------------------------------------------------------.
| Host | Status of TFA | PID   | Port | Version    | Build ID             |
+------+---------------+-------+------+------------+----------------------+
| rac1 | RUNNING       | 28335 | 5000 | 19.2.1.0.0 | 19210020190425110550 |
| rac2 | RUNNING       | 31169 | 5000 | 19.2.1.0.0 | 19210020190425110550 |
'------+---------------+-------+------+------------+----------------------'
Running Inventory in All Nodes…
Enabling Access for Non-root Users on rac1…
Adding default users to TFA Access list…
Summary of TFA Installation:
.--------------------------------------------------------------.
|                             rac1                             |
+---------------------+----------------------------------------+
| Parameter           | Value                                  |
+---------------------+----------------------------------------+
| Install location    | /u01/app/18.3.0/grid/tfa/rac1/tfa_home |
| Repository location | /u01/app/grid/tfa/repository           |
| Repository usage    | 0 MB out of 10240 MB                   |
'---------------------+----------------------------------------'
.--------------------------------------------------------------.
|                             rac2                             |
+---------------------+----------------------------------------+
| Parameter           | Value                                  |
+---------------------+----------------------------------------+
| Install location    | /u01/app/18.3.0/grid/tfa/rac2/tfa_home |
| Repository location | /u01/app/grid/tfa/repository           |
| Repository usage    | 0 MB out of 10240 MB                   |
'---------------------+----------------------------------------'
TFA is successfully installed…

Upgrading/Installing TFA with OSWatcher

The whole process is very simple and straightforward.
Post seems big but most of the content is a command output.

1. Download TFA Collector – TFA with Database Support Tools Bundle from Doc ID 1513912.1

2. Place downloaded zip file on rac1 and unzip it:

# cd /u01/app/sw
# ll
…
-rw-r--r-- 1 root root      264751391 Apr 25 19:18 TFA-LINUX_v19.2.1

# unzip TFA-LINUX_v19.2.1 

3. Install TFA:

[root@rac1 sw]# ./installTFA-LINUX 

TFA Installation Log will be written to File : /tmp/tfa_install_21556_2019_06_03-10_39_10.log
Starting TFA installation
 TFA Version: 192100 Build Date: 201904251105
 TFA HOME : /u01/app/12.2.0/grid/tfa/rac1/tfa_home
 Installed Build Version: 184100 Build Date: 201902262137
 TFA is already installed. Upgrading TFA
 TFA Upgrade Log : /u01/app/12.2.0/grid/tfa/rac1/tfapatch.log
 TFA will be upgraded on : 
 rac1
 rac2
 Do you want to continue with TFA Upgrade ? [Y|N] [Y]: Y
 Checking for ssh equivalency in rac2
 Node rac2 is not configured for ssh user equivalency
 SSH is not configured on these nodes : 
 rac2
 Do you want to configure SSH on these nodes ? [Y|N] [Y]: N
 Patching remote nodes using TFA Installer /u01/app/sw/installTFA-LINUX…
 Copying TFA Installer to rac2…
 lost connection
 Starting TFA Installer on rac2…
 Upgrading TFA on rac1 :
 Stopping TFA Support Tools…
 Shutting down TFA for Patching…
 Shutting down TFA
 Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.
 Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.
 Successfully shutdown TFA..
 No Berkeley DB upgrade required
 Copying TFA Certificates…
 Starting TFA in rac1…
 Starting TFA..
 Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Waiting up to 100 seconds for TFA to be started..
 . . . . . 
 Successfully started TFA Process..
 . . . . . 
 TFA Started and listening for commands
 Enabling Access for Non-root Users on rac1…
 Connection refused!rac2
 RemoteUtil : Connection refused!rac2
 .------------------------------------------------------------.
 | Host | TFA Version | TFA Build ID         | Upgrade Status |
 +------+-------------+----------------------+----------------+
 | rac1 |  19.2.1.0.0 | 19210020190425110550 | UPGRADED       |
 | rac2 | -           | -                    | NOT UPGRADED   |
 '------+-------------+----------------------+----------------'

The reason why it did not upgrade on rac2, is that I did not have ssh equivalency between nodes for root user.

I could enable ssh passwordless authentication and TFA would be upgraded in one step, but because of the security I will not enable it and just manually install TFA on the second node:

4. Copy installation file to rac2 and install:

[root@rac2 ~]# mkdir -p /u01/app/sw/
[root@rac2 ~]# chmod -R 777 /u01/app/sw/
[root@rac2 ~]# su - oracle
[oracle@rac2 ~]$ cd /u01/app/sw/  
[oracle@rac2 sw]$ scp rac1:/u01/app/sw/installTFA-LINUX .
installTFA-LINUX                           100%  254MB 114.9MB/s   00:02 
[root@rac2 sw]# ./installTFA-LINUX 
 TFA Installation Log will be written to File : /tmp/tfa_install_15370_2019_06_03-10_50_05.log
 Starting TFA installation
 TFA Version: 192100 Build Date: 201904251105
 TFA HOME : /u01/app/12.2.0/grid/tfa/rac2/tfa_home
 Installed Build Version: 184100 Build Date: 201902262137
 TFA is already installed. Upgrading TFA
 TFA Upgrade Log : /u01/app/12.2.0/grid/tfa/rac2/tfapatch.log
 TFA-00002 Oracle Trace File Analyzer (TFA) is not running
 TFA-00002 Oracle Trace File Analyzer (TFA) is not running
 Unable to determine the status of TFA in other nodes.
 TFA will be upgraded on Node rac2:
 Do you want to continue with TFA Upgrade ? [Y|N] [Y]: 
 Upgrading TFA on rac2 :
 Stopping TFA Support Tools…
 Shutting down TFA for Patching…
 Shutting down TFA
 Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.
 Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.
 . . . . . 
 . . . 
 Successfully shutdown TFA..
 No Berkeley DB upgrade required
 Copying TFA Certificates…
 Starting TFA in rac2…
 Starting TFA..
 Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Waiting up to 100 seconds for TFA to be started..
 . . . . . 
 Successfully started TFA Process..
 . . . . . 
 TFA Started and listening for commands
 Enabling Access for Non-root Users on rac2…
 .------------------------------------------------------------.
 | Host | TFA Version | TFA Build ID         | Upgrade Status |
 +------+-------------+----------------------+----------------+
 | rac2 |  19.2.1.0.0 | 19210020190425110550 | UPGRADED       |
 | rac1 |  19.2.1.0.0 | 19210020190425110550 | UPGRADED       |
 '------+-------------+----------------------+----------------'

5. Stop and Start TFA on rac1 and rac2:

# tfactl stop
Sending stoptfa
Success
Stopping TFA from the Command Line
Nothing to do !
LCM is not running
TFA is running  - Will wait 5 seconds (up to 3 times)  
TFA-00104 Cannot establish connection with TFA Server. Please check TFA Certificates
Killing TFA running with pid 16627
. . . 
Successfully stopped TFA..

# tfactl start
TFA-00002 Oracle Trace File Analyzer (TFA) is not running
Starting TFA..
Waiting up to 100 seconds for TFA to be started..
Successfully started TFA Process..

# ps -ef|grep OSWatcher
root     14528     1  0 May30 ?        00:44:11 /bin/sh ./OSWatcher.sh
root     14850 14528  0 May30 ?        00:00:58 /bin/sh ./OSWatcherFM.sh 48 /home/fg/oswbb/archive

6. Check OSWatcher repository location:

# ll /home/fg/oswbb/archive

drwxr-xr-x 2 root root  202 May 30 22:00 oswcpuinfo
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswifconfig
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswiostat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswmeminfo
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswmpstat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswnetstat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswnfsiostat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswpidstat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswpidstatd
drwxr-xr-x 2 root root    6 May 30 21:56 oswprvtnet
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswps
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswslabinfo
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswtop
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswvmstat
drwxr-xr-x 2 root root    6 May 30 21:56 oswxentop