Find 5 biggest files in Linux

I have used this command many times, but the interval between each usage is so big that I almost always forget the syntax.

So here it is:

# du -a / | sort -n -r | head -n 5

51190272	/
37705424	/root
33040524	/root/apache-tomcat-7.0.53
32802516	/root/apache-tomcat-7.0.53/logs
32802440	/root/apache-tomcat-7.0.53/logs/catalina.out

Reduce high CPU usage by TFA

Problem:

Cluster nodes experienced high CPU usage, after investigation one of the top CPU consumers on the server has been found to be a TFA process (2nd place):

 # Fri Feb 19 17:44:01 2021
AllCPU  OneCPU  PID     User    PR      NI      STime   RSS     Name
--------------------------------------------------------------------------------
11.75%  94.02%  23895   root    20      0       17:43   87M     ora_m001_ORCL2
1.42%   11.39%  2468    root    20      0       Feb02   736M    /opt/oracle.ahf/jre/bin/java -server -Xms256m -Xmx512m -Djava.awt.headless=true -Ddisable.checkForUpdate=true -XX:HeapDumpPath=/u01/app/oracle.ahf/data/rac02/diag/tfa -XX:ParallelGCThreads=5 oracle.rat.tfa.TFAMain /opt/oracle.ahf/tfa

Workaround:

In newer version of TFA, you can set CPU resource limit.

tfactl setresourcelimit 
 [-tool tool_name] 
 [-resource resource_type] 
 [-value value]

To limit TFA to a maximum of 50% of a single CPU, run the following:

# tfactl setresourcelimit -value 0.5

For more information, please check TFA official documentation.

If you don’t have newer version of TFA, you need to upgrade it first.

How to identify OS is Oracle Linux or RHEL?

There are several ways to identify that, I will suggest one of them using rpm -qf, that finds out what package a file belongs to:

Oracle Linux:

#  rpm -qf /etc/redhat-release
oraclelinux-release-7.8-1.0.7.el7.x86_64

RHEL:

# rpm -qf /etc/redhat-release
redhat-release-server-7.8-2.el7.x86_64

Boot in single user mode and rescue your RHEL7

Problem:

One of our customer incorrectly changed fstab file and rebooted the OS. As a result, VM was not able to start. Fortunately, cloud where this VM was located supported serial console.

Solution:

We booted in single user mode through serial console and reverted the changes back. To boot in single user mode and update necessary file, do as follows:

Connect to the serial console and while OS is booting in a grub menu press e to edit the selected kernel:

Find line that starts with linux16 ( if you don’t see it press arrow down ), go to the end of this line and type rd.break.

Press ctrl+x.

Wait for a while and system will enter into single user mode:

During this time /sysroot is mounted in read only mode, you need to remount it in read write:

switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot

You can revert any changes back by updating any file, in our case we updated fstab:

sh-4.2# vim /etc/fstab

You are a real hero, because you rescued your system!

pam_systemd(sshd:session): Failed to create session: Failed to activate service ‘org.freedesktop.login1’: timed out

Problem:

1. Slow ssh connections
2. System seems slow when trying to su to another user

/var/log/secure contains the following errors:

pam_systemd(sshd:session): Failed to create session: Failed to activate
service 'org.freedesktop.login1': timed out

Solution:

1. Restart systemd-logind service

# systemctl restart systemd-logind

2. Restart server

# reboot 

Note that the mentioned solutions are considered as temporary solutions (Frankly, I’ve never seen this error after restart. The problem happened with our two customers, who changed sshd_config file and did “something” after that, so the problem was caused by humman error in my all cases), for more information about this problem please see article at redhat site
https://access.redhat.com/discussions/3536621 .

Azure: yum install returns [Errno 14] curl#58 – “SSL peer rejected your certificate as expired.”

Action:

I have deleted tigervnc rpm and was trying to reinstall it but got the following error:

yum install tigervnc*

https://rhui-3.microsoft.com...x86_64/dotnet/1/debug/repodata/repomd.xml: [Errno 14] curl#58 - "SSL peer rejected your certificate as expired."
Trying other mirror.

Actually, the problem is not related to tigervnc only, it is global. During this time I was not able to run yum update or any installation using yum.

Causes:

Red Hat Update Infrastructure (RHUI) certificate has expired and it needs to be updated.

Solution:

Update RHUI certificate using the following rpm:

For RHEL 7:

# curl -o azureclient.rpm https://rhui-1.microsoft.com/pulp/repos/microsoft-azure-rhel7/rhui-azure-rhel7-2.2-74.noarch.rpm
# sudo rpm -U azureclient.rpm

For RHEL 6:

# curl -o azureclient.rpm https://rhui-1.microsoft.com/pulp/repos/microsoft-azure-rhel6/rhui-azure-rhel6-2.2-74.noarch.rpm
# sudo rpm -U azureclient.rpm

Linux: Rename files from uppercase to lowercase

If you have downloaded Oracle 18c installation files, you may need to change downloaded file  names from uppercase letters into lowercase. 🙂

[root@rac1 ~]# cd /sw
[root@rac1 sw]# for i in LINUX.X64_180000_*; do mv $i `echo $i |tr [:upper:] [:lower:]`; done

You may think these are just two files and why I need script? I can do it manually.. 🙂
You are right , but scripting is much more fun. Good luck!

Change timezone on Linux using /etc/localtime file

Assume we have timezone UTC:

[root@oradbfw-hq01-mgmt ~]# date
Mon Sep 11 09:29:27 UTC 2017

On many distributions , like OEL, CentOS.. . the timezone is conntrolled by localtime file located in /etc directory.

# cd /etc

Localtime is a symbolic link , so we can remove it

# rm localtime

All timezones are located under /usr/share/zoneinfo.
I live and my servers are located in Tbilisi, Georgia. So searching my timezone file:

[root@oradbfw-hq01-mgmt etc]# ll /usr/share/zoneinfo/Asia/Tbilisi
-rw-r–r–. 1 root root 1024 Mar 24 09:21 /usr/share/zoneinfo/Asia/Tbilisi

My country is not in Asia, but in Europe 🙂 this condition changed recently so Linux has not updated and still thinks we in Asia. (Don’t tell this to my goverment 🙂 )

Let’s create correct symbolic link:

# cd /etc
# ln -s /usr/share/zoneinfo/Asia/Tbilisi localtime

Let’s check that it was changed:

[root@oradbfw-hq01-mgmt etc]# date
Mon Sep 11 13:47:04 +04 2017

Create a Local Yum Repository for Oracle Linux 6

0. Just to note –repoid parameter has two dashes in front (- – repoid). Unfortunately wordpress makes it one dash. Please, do not blame me 🙂

1. Create OEL.repo file in /etc/yum.repos.d and add the following entry

# vi /etc/yum.repos.d/OEL.repo

[oraclerpms]
name=OracleEnterpriseLinux
baseurl=http://yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
enabled=1

2. Install yum-utils and createrepo packages

# yum install yum-utils createrepo

3. It is better to add another disk and create repository there, do not use / disk(disk where OS is installed) for repository. It is just a best practice.

Create necessary folders:

# mkdir -p /OEL_repo/OEL6/latest
# mkdir -p /OEL_repo/logs
# mkdir -p /OEL_repo/scripts

4. Download rpms.
If you are using proxy you may need to set proxy parameters like  export http_proxy=http://proxy_server_ip:proxy_port/

/usr/bin/reposync –newest-only –repoid=oraclerpms -p /OEL_repo/OEL6/latest

5. After download completes create repository on downloaded rpms:

/usr/bin/createrepo /OEL_repo/OEL6/latest/oraclerpms/getPackage/

####################
The above steps should be done every time you want to update your local repository, so let’s create script and place all them in one file.

vi /OEL_repo/scripts/repo_sync.sh

#!/bin/bash

LOG_FILE=/OEL_repo/logs/repo_sync_$(date +%Y.%m.%d).log

# Delete old logs
find /OEL_repo/logs/repo_sync* -mtime +5 -delete; >> $LOG_FILE 2>&1

#Clean cache
yum clean all

# Sync repositories
/usr/bin/reposync /usr/bin/reposync –newest-only –repoid=oraclerpms -p /OEL_repo/OEL6/latest >> $LOG_FILE 2>&1

/usr/bin/createrepo /OEL_repo/OEL6/latest/oraclerpms/getPackage/ >> $LOG_FILE 2>&1

2. Change permissions for the executable script:

# chmod u+x /OEL_repo/scripts/repo_sync.sh

3. Set up a CRON job to run the script on a daily basis. The following entry runs the script each day at 01:00.

0 1 * * * /OEL_repo/scripts/repo_sync.sh > /dev/null 2>&1

###############

1. Install the Apache HTTP servers, start it and make sure it restarts automatically on reboot.

# yum install httpd
# service httpd start
# chkconfig httpd on

2. Create directory and place link there

mkdir -p /var/www/html/OEL_repo/OEL6/latest
ln -s /OEL_repo/OEL6/latest/oraclerpms/getPackage/ /var/www/html/OEL_repo/OEL6/latest/x86_64

# cd /var/www/html
# wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Note: /var/www/html is the default DocumentRoot directory. To check yours(sysadmin may have changed it) please check /etc/httpd/conf/httpd.conf file and find the keyword DocumentRoot

##############

1. Configure clients

vi /etc/yum.repos.d/local-ol6.repo

[oraclerpms]
name=Oracle Linux $releasever Latest ($basearch)
baseurl=http://RepositoryServerHostname.domain.ge/OEL_repo/OEL6/latest/$basearch/
gpgkey=http://RepositoryServerHostname.domain.ge/RPM-GPG-KEY-oracle-ol6
gpgcheck=1
enabled=1

Now you are able to install packages on clients by

# yum install ntp

Upgrade Oracle Database from 11g to 12c

  1. Download Oracle 12c software fromhttp://www.oracle.com/technetwork/database/enterprise-edition/downloads/database12c-linux-download-1959253.html

    unzip files:

    unzip linuxamd64_12102_database_1of2.zip
    unzip linuxamd64_12102_database_2of2.zip
  2. Make another home for 12c.
    mkdir -p /u01/app/oracle/product/12.1.0/dblb
  3. Change permissions for /u01 directory.  If 11g home is also located in /u01 you have already done the steps bellow. Just check that permissions are the following:
    chown -R oracle:oinstall /u01
    chmod -R 775 /u01
  4. Change the following parameters in response file, other parameters just leave blank.
    Response file is located in installation directory… extract_drectory/database/response/db_install.rsp

     oracle.install.option=INSTALL_DB_SWONLY
     ORACLE_HOSTNAME=DBServerHostname
     UNIX_GROUP_NAME=oinstall
     INVENTORY_LOCATION=/u01/app/oraInventory
     SELECTED_LANGUAGES=en
     ORACLE_HOME=/u01/app/oracle/product/12.1.0/dblb
     ORACLE_BASE=/u01/app/oracle
     oracle.install.db.InstallEdition=EE
     oracle.install.db.DBA_GROUP=dba
     oracle.install.db.OPER_GROUP=dba
     oracle.install.db.BACKUPDBA_GROUP=dba
     oracle.install.db.DGDBA_GROUP=dba
     oracle.install.db.KMDBA_GROUP=dba
     SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
     DECLINE_SECURITY_UPDATES=true
  5. Go to the 12c installation folder  and run runInstaller, to install 12c home:
     ./runInstaller -silent -responseFile /install/database/response/db_install.rsp  -waitforcompletion -showProgress

    When it asks , connect to the server via root user and run

    /u01/app/oracle/product/12.1.0/dblb/root.sh
  6. At this time your database should be turned on from 11g home. Connect to the database via SYS user and run the following scripts: emremove.sql will remove EM repository. olspreupgrade.sql will run preupgrade scripts

    You should also purge the recyclebin. For reducing upgrade time.

    Note: these scripts should be run to the open database , that is turned on by 11g

    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dblb
     sqlplus / as sysdba
     @/u01/app/oracle/product/12.1.0/dblb/rdbms/admin/emremove.sql
     @/u01/app/oracle/product/12.1.0/dblb/rdbms/admin/olspreupgrade.sql
     purge recyclebin;
  7. Run DBUA from 12c home to upgrade existing database
    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dblb
    /u01/app/oracle/product/12.1.0/dblb/bin/dbua -silent \
    -sid ORCL \
    -oracleHome /u01/app/oracle/product/11.2.0/dblb \
    -diagnosticDest /u01/app/oracle \
    -recompile_invalid_objects true \
    -degree_of_parallelism 40 \
    -upgradeTimezone \
    -emConfiguration NONE \
    -keepHiddenParams \
    -gatheringStatistics \
    -upgrade_parallelism 40
  8. To check that everything was upgraded successfully, after successful message from the previous command, check the following:
    cat /etc/oratab
    ORCL:/u01/app/oracle/product/12.1.0/dblb:N
    cat /u01/app/oraInventory/ContentsXML/inventory.xml
     <VERSION_INFO>
     <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
     <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
     </VERSION_INFO>

    Update the following parameters in your  .bash_profile:

     export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dblb
     export LD_LIBRARY_PATH=/u01/app/oracle/product/12.1.0/dblb/lib
  9. Network file configuration. Stop listener that was previously started from 11g. Move listener.ora and tnsnames.ora files to 12c home and rename them in old location. Start the listener from 12c home.
    /u01/app/oracle/product/11.2.0/dblb/bin/lsnrctl stop
    cp /u01/app/oracle/product/11.2.0/dblb/network/admin/listener.ora /u01/app/oracle/product/12.1.0/dblb/network/admin
    cp /u01/app/oracle/product/11.2.0/dblb/network/admin/tnsnames.ora /u01/app/oracle/product/12.1.0/dblb/network/admin
    mv /u01/app/oracle/product/11.2.0/dblb/network/admin/tnsnames.ora /u01/app/oracle/product/11.2.0/dblb/network/admin/tnsnames.ora.old
    mv /u01/app/oracle/product/11.2.0/dblb/network/admin/listener.ora /u01/app/oracle/product/11.2.0/dblb/network/admin/listener.ora.old
    /u01/app/oracle/product/12.1.0/dblb/bin/lsnrctl start
  10. Connect to the database using 12c home and check again the version in v$instance view:
    export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dblb
    sqlplus / as sysdba
    select instance_name,version,status
    from v$instance;
  11. If you want to deinstall 11g home run the following :
    /u01/app/oracle/product/11.2.0/dblb/deinstall/deinstall