Linux STRESS command usage example


During high CPU usage in kernel space we have noticed brownouts on our database nodes. For finding the reason of the problem we wanted to reproduce the issue and somehow trigger high %sy usage on our nodes.

I have found stress tool very useful and want to share my experience with you.


1. Install stress tool via yum:

# yum install stress

2. Stress has several options to use:

[root@rac1 ~]# stress

`stress' imposes certain types of compute stress on your system

Usage: stress [OPTION [ARG]] ...
 -?, --help         show this help statement
     --version      show version statement
 -v, --verbose      be verbose
 -q, --quiet        be quiet
 -n, --dry-run      show what would have been done
 -t, --timeout N    timeout after N seconds
     --backoff N    wait factor of N microseconds before work starts
 -c, --cpu N        spawn N workers spinning on sqrt()
 -i, --io N         spawn N workers spinning on sync()
 -m, --vm N         spawn N workers spinning on malloc()/free()
     --vm-bytes B   malloc B bytes per vm worker (default is 256MB)
     --vm-stride B  touch a byte every B bytes (default is 4096)
     --vm-hang N    sleep N secs before free (default none, 0 is inf)
     --vm-keep      redirty memory instead of freeing and reallocating
 -d, --hdd N        spawn N workers spinning on write()/unlink()
     --hdd-bytes B  write B bytes per hdd worker (default is 1GB)

Example: stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M --timeout 10s

Note: Numbers may be suffixed with s,m,h,d,y (time) or B,K,M,G (size).

To cause high %sy you need to use –vm option and find appropriate number of workers, in my case 50 workers were enough to cause an issue.

In the following example, stress will run 50 workers and timeout for the run will be 200s:

# stress --vm 50 --timeout 200s

From another terminal tab, run top command to monitor %sy usage (81.2%) :

See short video demonstration below:

Flashgrid CHM and Basic Troubleshooting Part 2

Flashgrid CHM and Basic Troubleshooting Part 1

Add new virtual machine in VBox and install Oracle Linux


This blog post belongs to my student at Business and Technology University Ivane Metreveli, thank you Ivane for participating in this project.

  1. First of all, you need to download Oracle Linux iso file from or from After that, run VirtualBox, click New button and create new virtual machine:

2. Set Name of the Virtual Machine and select operation system as follows, click Next

3. Select appropriate RAM amount, 3GB RAM is recommended for normal processing, click on Next button and jump to next step

4. Now, Select Create a virtual hard disk now option and click Create button

5. Select VDI(virtualbox Disk image)

6. Select Dynamically allocated if you don’t want take hard disk space immediately

7. Select file size (disk size for VB) and the location, click Create button to finish virtual machine creation process

8. Virtual machine is already is created. Before we open/start VM, we load iso file in the machne, click Settings and follow me

9. Navigate to Storage and click CD icon,  on the right side of the window, under attributes, click CD icon and add virtual machine’s .iso file.

10. After that, you can click start button

11. Select .iso files or click folder icon and open folder where .iso file is located, select it and click start

12. Next step is OS installation process, here you select Install Oracle linux 7.6 and click enter to start installation process:

13. Select system language and click continue

14. Select installation destiantion

15. Select the disk where you want to install system. You can select virtual disk, that you have created in the previous step or add a new one. Select disk and click Done button;

16. Now all parameter is ready. Click Begin Installation and wait for finishing the process

17. Set password and click Done

18. Installation is in progress, need to wait more

19. Installation proess is finished, click Roboot button and move to the next step:

20. Installation is finised now, you can start working with Oracle Linux:

Install Linux in Virtual Box


This blog post belongs to my student at Business and Technology University Saba Lapanashvili, thank you Saba for participating in this project.


– VirtualBox
– Linux iso file ( For example Linux Mint 15 )

Step 1: Choose System Type

– After installing VirtualBox, click New
– Fill the Name field: e.g Linux Mint 15
– Select Type: Linux
– Select Version: Ubuntu

Step 2: Select the Amount of RAM

– Select the amount of RAM, e.g 2048 MB = 2 GB

Step 3: Configure Hard Disk Settings

– Choose Create a virtual hard drive now, to make a virtual disk space
– Select the VDI 
– Choose Dynamically allocated
– Select the amount of hard drive size

Step 4: Choose Linux ISO File

Now we have done hardware settings

– Click Start to launch system
– Choose your system iso file from your computer, for example my system iso file is (linuxmint-15-cinnamon-dvd-64bit.iso)

Step 5: Install Linux and Make Account

– Click on the Install Linux Mint
– And select Erase disk and install Linux Mint
– Then press Install Now
– Make your account
– Press Continue

Step 6: Congratulations

Congratulations now you have Linux on your Windows.

Python: ImportError: No module named typing


Installing joblib module is throwing error:

# pip install joblib
ImportError: No module named typing

Trying to install typing, returns the same error:

# pip install typing
ImportError: No module named typing


Install python-typing using yum:

# yum install python-typing

Install azcli on Linux

0. Import gpgkey

$ sudo rpm --import
  1. Create local azure-cli repository information.
# sh -c 'echo -e "[azure-cli]
name=Azure CLI
gpgkey=" > /etc/yum.repos.d/azure-cli.repo'

2. Install with the yum install command.

$ yum install azure-cli

3. Run the login command.

$ az login

To sign in, use a web browser to open the page and enter the code MYCODEHERE to authenticate.

4. Go to and enter the mentioned code

5. Click Next -> choose your account.

Python: Manipulation on file content located in s3 archived as tar.gz without downloading


Need to analyze several values from multiple files that are archived as tar.gz and located on s3. This operation must be performed without downloading or extracting tar.gz

HARBOR: I am neither the Python expert nor the developer, so it is assumed that I am having mistakes in it or script could be written shorter and easier way than I did.
But it satisfies my needs. So please use it as an example only and investigate the content of it.

Hierarcy of the tar.gz file is the following (sample):


  1. Create file with the following content and grant executable permission to that file:

    Note: Update the following entries in the file according to your environment.
AWS_ACCESS_KEY_ID = "my key goes here"    
AWS_SECRET_ACCESS_KEY = "my secret key goes here"

Content of

import boto3
import tarfile
import joblib
import io
import sys

class S3Loader(object):
    AWS_ACCESS_KEY_ID = "my key goes here"
    AWS_SECRET_ACCESS_KEY = "my secret key goes here"
    AWS_REGION_NAME = "us-east-1"
    AWS_STORAGE_BUCKET_NAME = "flashgrid-support"
    def __init__(self):
        self.s3_client = boto3.client("s3",

    def load_tar_file_s3_into_object_without_download(self, s3_filepath):

        # Describing variables search pattern
        match = ("Disk latency above threshold")
        notmatch = (".lun")

        s3_object = self.s3_client.get_object(Bucket=self.AWS_STORAGE_BUCKET_NAME, Key=s3_filepath)
        wholefile = s3_object['Body'].read()
        fileobj = io.BytesIO(wholefile)

        # Opening first tar.gz file
        tar =

        # Searching nested tar.gz files
        childgz = [ for f in tar.getmembers() if'.gz')]

        # Extracting file named flashgrid_cluster which is located in the first tar.gz
        node1gz =[0]))
        fgclustername = [ for f in node1gz.getmembers() if'flashgrid_cluster')]
        fgclusternamecontent = node1gz.extractfile(fgclustername[0])

        # Extracting text that contains string "Cluster Name:"
        for fgclusternameline in fgclusternamecontent:
           if "Cluster Name:" in fgclusternameline:
#        print(len(childgz))
#        print(clustername)
#        print(childgz)
#        nodegzlist=list('')
#        nodemonfilelist=list('')

# Extracting file node_monitor_error.log from all nested tar.gz files
        for i in childgz:
#          nodegzlist.append(
           cur_gz_file_extracted =
#           print(
           cur_node_mon_file = [ for f in cur_gz_file_extracted.getmembers() if'node_monitor-error.log')]

# Path to node_monitor_error.log contains hostname inside so extracting string that is the hostname
           cur_node_name = cur_node_mon_file[0].split("/")[0]
#           print(cur_node_name)
#           nodemonfilelist.append([ for f in curfile.getmembers() if'node_monitor-error.log')])
#           print(nodemonfilelist[0],nodemonfilelist[1],nodemonfilelist[2])

# Extracting content of node_monitor_error.log file
           cur_node_mon_file_content = cur_gz_file_extracted.extractfile(cur_node_mon_file[0])
#           print(cur_node_mon_file_content)
#        fgclusternamecontent = node1gz.extractfile(fgclustername[0])

#        for fgclusternameline in fgclusternamecontent:
#           if "Cluster Name:" in fgclusternameline:
#             clustername=fgclusternameline

# Selecting lines from the extracted file and filtering based on match criteria (match, notmatch variables)
           for cur_node_mon_file_content_line in cur_node_mon_file_content:
            if match in cur_node_mon_file_content_line and not (notmatch in cur_node_mon_file_content_line):
               # Extracting time from the string, knowing the exact position
               time = cur_node_mon_file_content_line.split(" ")[0] + " " + cur_node_mon_file_content_line.split(" ")[1]
               cur_node_mon_file_line_splitted = cur_node_mon_file_content_line.split(" ")
               # Extracting necessary values after spliting the content by delimiter " "
               print(clustername.strip(),cur_node_name,cur_node_mon_file_line_splitted[8] , time,  cur_node_mon_file_line_splitted[17] + " " + cur_node_mon_file_line_splitted[18].strip())
#               print(nodemonfileline)

if __name__ == "__main__":
    s3_loader = S3Loader()

     # Script takes 1 argument


2. Run .py file and pass path of the tar.gz file

# ./ "mybucketname/myfoldername/myfile.tar.gz"

So the search is happening for flashgrid_cluster and node_monitor_error.log file content, for which two nested tar.gz should be analyzed.

RMAN restore on ASM fails ORA-12547: TNS:lost contact, WARNING: ASMB0 exiting with error


Our client was not able to restore controlfile on +FRA diskgroup:

RMAN> restore controlfile to '+FRA' from '<backup file location>';
RMAN-03002 failure of restore command at 2021/05/24 19:12:19
ORA-19870: error while restoring backup piece 
ORA-19504: failed to create file '+FRA'
ORA-17502: ksfdcre:4 Failed to create file +FRA
ORA-15001: diskgroup "FRA" does not exist or is not mounted
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact


oracle binary under GI home did not have correct permissions:

[grid@rac1 bin]$ ll oracle
-rwxr-x--x 1 grid oinstall 420332360 Oct 13  2020 oracle


Change permissions to 6751 and retry the restore:

[grid@rac1 bin]$ chmod 6751 oracle

[grid@rac1 bin]$ ll oracle
-rwsr-s--x 1 grid oinstall 420332360 Oct 13  2020 oracle

rpm -qa gets thread died in Berkeley DB library


After checking if flashgrid-clan package was installed, got this error:

error: rpmdb: BDB0113 Thread/process 2884/140438918064192 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 - (-30973)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: BDB0113 Thread/process 2884/140438918064192 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm
package flashgrid-clan is not installed


If you see rpmdb errors during package management (rpm, yum), it means that the RPM database is corrupted.


# mkdir /var/lib/rpm/backup
# cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/
# rm -f /var/lib/rpm/__db.[0-9][0-9]*
# rpm --quiet -qa
# rpm --rebuilddb
# yum clean all