Boot in single user mode and rescue your RHEL7

Problem:

One of our customer incorrectly changed fstab file and rebooted the OS. As a result, VM was not able to start. Fortunately, cloud where this VM was located supported serial console.

Solution:

We booted in single user mode through serial console and reverted the changes back. To boot in single user mode and update necessary file, do as follows:

Connect to the serial console and while OS is booting in a grub menu press e to edit the selected kernel:

Find line that starts with linux16 ( if you don’t see it press arrow down ), go to the end of this line and type rd.break.

Press ctrl+x.

Wait for a while and system will enter into single user mode:

During this time /sysroot is mounted in read only mode, you need to remount it in read write:

switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot

You can revert any changes back by updating any file, in our case we updated fstab:

sh-4.2# vim /etc/fstab

You are a real hero, because you rescued your system!

PRVG-11069 : IP address “169.254.0.2” of network interface “idrac” on the node “primrac1” would conflict with HAIP usage

Problem:

Oracle 18c GI configuration precheck was failing with the following error:

Summary of node specific errors 

primrac2  - PRVG-11069 : IP address "169.254.0.2" of network interface "idrac" on the node "primrac2" would conflict with HAIP usage.  
- Cause:  One or more network interfaces have IP addresses in the range (169.254..), the range used by HAIP which can create routing conflicts.  
- Action:  Make sure there are no IP addresses in the range (169.254..) on any network interfaces. 

primrac1  - PRVG-11069 : IP address "169.254.0.2" of network interface "idrac" on the node "primrac1" would conflict with HAIP usage. 
- Cause:  One or more network interfaces have IP addresses in the range (169.254..), the range used by HAIP which can create routing conflicts.  
- Action:  Make sure there are no IP addresses in the range (169.254..) on any network interfaces.  

On each node additional network interface – named idrac was started with the ip address 169.254.0.2. I tried to set static ip address in /etc/sysconfig/network-scripts/ifcfg-idrac , also tried to bring the interface down – but after some time interface was starting up automatically and getting the same ip address.

Cluster nodes were DELL servers with Dell Remote Access Controller(iDRAC) Service Module installed. For more information about this module installation/deinstallation… can be found here https://topics-cdn.dell.com/pdf/idrac-service-module-v32_users-guide_en-us.pdf

Servers were configured by system administrator and was not clear why this module was there, we are not using iDRAC module and the only option that we had was to remove/uninstall that module. (configuring module should also be possible to avoid such situation, but we keep our servers as clean as possible without having unsed services)

Solution:

Uninstalled iDRAC module (also expained in the above pdf):

# rpm -e dcism 

After uninstalling it idrac interface did not started anymore, so we could continue GI configuration.

Azure: yum install returns [Errno 14] curl#58 – “SSL peer rejected your certificate as expired.”

Action:

I have deleted tigervnc rpm and was trying to reinstall it but got the following error:

yum install tigervnc*

https://rhui-3.microsoft.com...x86_64/dotnet/1/debug/repodata/repomd.xml: [Errno 14] curl#58 - "SSL peer rejected your certificate as expired."
Trying other mirror.

Actually, the problem is not related to tigervnc only, it is global. During this time I was not able to run yum update or any installation using yum.

Causes:

Red Hat Update Infrastructure (RHUI) certificate has expired and it needs to be updated.

Solution:

Update RHUI certificate using the following rpm:

For RHEL 7:

# curl -o azureclient.rpm https://rhui-1.microsoft.com/pulp/repos/microsoft-azure-rhel7/rhui-azure-rhel7-2.2-74.noarch.rpm
# sudo rpm -U azureclient.rpm

For RHEL 6:

# curl -o azureclient.rpm https://rhui-1.microsoft.com/pulp/repos/microsoft-azure-rhel6/rhui-azure-rhel6-2.2-74.noarch.rpm
# sudo rpm -U azureclient.rpm

Downloading Oracle files on Linux via wget

There are several ways to download files from Oracle site.

We will use one of the methods to download Oracle Proactive Bundle Patch on the Linux machine.

First of all, find the desired file and copy its link address:

Run WGET by passing the following parameters:

# wget --http-user=mariam.kupa@gmail.com --ask-password  "https://updates.oracle.com/Orion/Services/download/p27968010_121020_Linux-x86-64.zip?aru=22331652&patch_file=p27968010_121020_Linux-x86-64.zip" -O p27968010_121020_Linux-x86-64.zip
Password:

That’s it!

 

Script to capture and restore file permissions

Backing up file permissions is the best practice. Even extra permissions on files can mess up installed software.

Editing this post:

Thanks to zhwsh about this comment, that even does not need to be explained:

“getfacl -R /u01/app/11.2.0.4/grid > dir_privs.txt
setfacl –restore dir_privs.txt”

In front of the “restore”, you should write two hyphens “- -”

In any case leaving perl script that does the same as getfacl. 

Usage:

chmod 755 backup_permissions.pl

./backup_permissions.pl <Path>

Script: 

#!/usr/bin/perl -w
#
# Captures file permissions and the owner of the files
# useage : perm1.pl <path to capture permission>
#

use strict;
use warnings;
use File::Find;
use POSIX();

my (@dir) = @ARGV;
my $linecount=0 ;

#print @ARGV, $#ARGV;

if ($#ARGV < 0) {
print “\n\nOpps….Invalid Syntax !!!!\n” ;
print “Usage : ./perm1.pl <path to capture permission>\n\n” ;
print “Example : ./perm1.pl /home/oralce\n\n” ;
exit ;
}
my $logdir=$dir[0] ;
#my ($sec, $min, $hr, $day, $mon, $year) = localtime;
##my ($dow,$mon,$date,$hr,$min,$sec,$year) = POSIX::strftime( ‘%a %b %d %H %M %S %Y’, localtime);
my $date = POSIX::strftime( ‘%a-%b-%d-%H-%M-%S-%Y’, localtime);
my $logfile=”permission-“.$date;
my $cmdfile=”restore-perm-“.$date.”.cmd” ;

open LOGFILE, “> $logfile” or die $! ;
open CMDFILE, “> $cmdfile” or die $! ;
find(\&process_file,@dir);

print “Following log files are generated\n” ;
print “logfile : “.$logfile. “\n” ;
print “Command file : “.$cmdfile. “\n” ;
print “Linecount : “.$linecount.”\n” ;
close (LOGFILE) ;
close (CMDFILE) ;

sub process_file {
my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks,$username,$user,$pass,$comment,$home,$shell,$group);
my %uiduname = () ;
my %gidgname = () ;
my $filename = $File::Find::name;

#### Building uid, username hash

open (PASSWDFILE, ‘/etc/passwd’) ;

while ( <PASSWDFILE>) {
($user,$pass,$uid,$gid,$comment,$home,$shell)=split (/:/) ;
$uiduname{$uid}=$user ;
}
close (PASSWDFILE) ;

#### Building gid, groupname hash

open (GRPFILE, ‘/etc/group’) ;

while ( <GRPFILE>) {
($group,$pass,$gid)=split (/:/) ;
$gidgname{$gid}=$group ;
}
close (GRPFILE) ;

($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks) = stat(“$filename”);
# printf “%o %s %s %s\n”, $mode & 07777, $uiduname{$uid}, $gidgname{$gid}, $filename ;
printf LOGFILE “%o %s %s %s\n”, $mode & 07777, $uiduname{$uid}, $gidgname{$gid}, $filename ;
printf CMDFILE “%s %s%s%s %s\n”, “chown “,$uiduname{$uid}, “:”, $gidgname{$gid}, $filename ;
printf CMDFILE “%s %o %s\n”, “chmod “,$mode & 07777, $filename ;
# printf “%o %s %s %s\n”, $mode & 07777, $uiduname{$uid}, $gidgname{$gid}, $filename ;
$linecount++ ;
}

Note:

The above script generates restore-perm-<timestamp>.cmd file.

When you want to restore permissions make this file executable and run:

chmod 755 restore-perm-<timestamp>.cmd

./restore-perm-<timestamp>.cmd

 

 

 

Change timezone on Linux using /etc/localtime file

Assume we have timezone UTC:

[root@oradbfw-hq01-mgmt ~]# date
Mon Sep 11 09:29:27 UTC 2017

On many distributions , like OEL, CentOS.. . the timezone is conntrolled by localtime file located in /etc directory.

# cd /etc

Localtime is a symbolic link , so we can remove it

# rm localtime

All timezones are located under /usr/share/zoneinfo.
I live and my servers are located in Tbilisi, Georgia. So searching my timezone file:

[root@oradbfw-hq01-mgmt etc]# ll /usr/share/zoneinfo/Asia/Tbilisi
-rw-r–r–. 1 root root 1024 Mar 24 09:21 /usr/share/zoneinfo/Asia/Tbilisi

My country is not in Asia, but in Europe 🙂 this condition changed recently so Linux has not updated and still thinks we in Asia. (Don’t tell this to my goverment 🙂 )

Let’s create correct symbolic link:

# cd /etc
# ln -s /usr/share/zoneinfo/Asia/Tbilisi localtime

Let’s check that it was changed:

[root@oradbfw-hq01-mgmt etc]# date
Mon Sep 11 13:47:04 +04 2017

error: package cvuqdisk is not installed

I have applied patch on RAC and after running postinstall script on the first node, it failed because of some file permission and the problem started…

I could not startup clusterware on first node.

I have deconfigured clusterware by:

[root@lbdm01-dr-adm ~]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose

And here I got error(cvuqdisk):

PRCR-1070 : Failed to check if resource ora.net1.network is registered
CRS-0184 : Cannot communicate with the CRS daemon.
PRCR-1070 : Failed to check if resource ora.helper is registered
CRS-0184 : Cannot communicate with the CRS daemon.
PRCR-1070 : Failed to check if resource ora.ons is registered
CRS-0184 : Cannot communicate with the CRS daemon.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘lbdm01-dr-adm’
CRS-2679: Attempting to clean ‘ora.cssd’ on ‘lbdm01-dr-adm’
CRS-2680: Clean of ‘ora.cssd’ on ‘lbdm01-dr-adm’ failed
CRS-2799: Failed to shut down resource ‘ora.cssd’ on ‘lbdm01-dr-adm’
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on ‘lbdm01-dr-adm’ has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
2016/09/26 19:54:12 CLSRSC-463: The deconfiguration or downgrade script could not stop current Oracle Clusterware stack.

2016/09/26 19:54:12 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2016/09/26 19:54:26 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

error: package cvuqdisk is not installed
2016/09/26 19:54:26 CLSRSC-557: Oracle Clusterware stack on this node has been successfully deconfigured. There were some errors which can be ignored.

 

I have searched information about this package on documentation and I have found the following:

Link: https://docs.oracle.com/database/121/LADBI/pre_install.htm#LADBI7632

Installing the cvuqdisk RPM for Linux

If you do not use an Oracle Preinstallation RPM, then you must install the cvuqdisk RPM. Without cvuqdisk, the Cluster Verification Utility cannot find shared disks, and you receive a “Package cvuqdisk not installed” error when you run the Cluster Verification Utility. Use the cvuqdisk RPM for your hardware (for example, x86_64, or i386).

To install the cvuqdisk RPM, complete the following procedure:

  1. Locate the cvuqdisk RPM package, which is in the directory rpm on the Oracle Database installation media. If you installed Oracle Grid Infrastructure, then it is in the directory oracle_home1/cv/rpm.
  2. Log in as root.
  3. Use the following command to find if you have an existing version of the cvuqdisk package:
  4.  
    # rpm -qi cvuqdisk
    

    If you have an existing version, then enter the following command to deinstall the existing version:

    # rpm -e cvuqdisk
    
  5. Set the environment variable CVUQDISK_GRP to point to the group that owns cvuqdisk, typically oinstall, for example:
    # CVUQDISK_GRP=oinstall; export CVUQDISK_GRP
    
  6. In the directory where you have saved the cvuqdisk RPM, use the following command to install the cvuqdisk package:
    rpm -iv package
    

    For example:

    # rpm -iv cvuqdisk-1.0.9-1.rpm

So I have found the mentioned package in the following directory and installed it.

cd /u01/app/12.1.0.2/grid/cv/rpm

yum install cvuqdisk-1.0.9-1.rpm

Note: The problem is strange but, I am not writing why this happened in this post, because I don’t know it yet 🙂
The aim of this post is that you should know where to find cvuqdisk package and what is it for 🙂

Good Luck!

Creating a Snapshot-Based Backup of Oracle Linux Database Server(Exadata)

Story: We have Exadata x5-2 servers and they must be PCI compliant. Our PCI scanner found a lot of vulnerabilities on these machines, most of them were rpm upgrades.

So we are in a hell (devil) . Exadata is not a toy and upgrading it’s system is not like playing :):)

We must backup the system , and take the image before doing such things.

Because of Exadata has LVM-s installed we can use LVM based backups.

So let’s start:

Creating a Snapshot-Based Backup of Oracle Linux Database Server with Uncustomized Partitions

This means that you have not changed anything after the initial installation.

a) create mount point where you save backups. It is better if you use NFS. But if you don’t have NFS you may temporarily use local storage and then copy backup to another disk to be more safe.

Exadata local disk is 1.6TB in size and there is about 1.4TB free , so we can create another logical volume for backup. Let’s create logical volume 400GB  in size.
Here, I can tell you the difference between DBA and Sysadmin 🙂 DBA needs more and more space and always takes more than required for being calm for next 5 years. And sysadmin is giving to DBA less space than required and everytime you ask him/her storage his/her face becomes like thinking_80_anim_gif

# lvcreate -L 400GB -n backup VGExaDb
# mkdir /backup
# mkfs.ext4 /dev/VGExaDb/backup
# echo “/dev/VGExaDb/backup     /backup                 ext4    defaults        0 0” >> /etc/fstab
# mount /backup

b) Take a snapshot-based backup of the / (root), /u01. Name them root_snap and u01_snap. Mount them to the newly created directories called exadata_os_backup and exadata_u01_backup.

# mkdir /backup/exadata_os_backup
# mkdir /backup/exadata_u01_backup
# lvcreate -L1G -s -c 32K -n root_snap /dev/VGExaDb/LVDbSys1
Logical volume “root_snap” created.
# e2label /dev/VGExaDb/root_snap DBSYS_SNAP
# mount /dev/VGExaDb/root_snap /backup/exadata_os_backup -t ext4
# lvcreate -L5G -s -c 32K -n u01_snap /dev/VGExaDb/LVDbOra1
Logical volume “u01_snap” created.
# e2label /dev/VGExaDb/u01_snap DBORA_SNAP
# mount /dev/VGExaDb/u01_snap  /backup/exadata_u01_backup -t ext4

c)  Create the backup file using the following command:

# cd  /backup
# tar -pjcvf /backup/os_backup.tar.bz2 * –exclude os_backup.tar.bz2 > /backup/os_backup.stdout 2> /backup/os_backup.stderr

Check the /tmp/backup_tar.stderr file for any significant errors. Errors about failing to tar open sockets, and other similar errors, can be ignored.

d) Unmount the snapshots and remove the snapshots for the root and /01 directories using the following commands:

umount /backup/exadata_u01_backup
umount /backup/exadata_os_backup
lvremove /dev/VGExaDb/u01_snap
lvremove /dev/VGExaDb/root_snap

More information:
https://docs.oracle.com/cd/E50790_01/doc/doc.121/e51951/db_server.htm#DBMMN21380
There is also discussed “Creating a Snapshot-Based Backup of Oracle Linux Database Server with Customized Partitions”.

 

Create a Local Yum Repository for Oracle Linux 6

0. Just to note –repoid parameter has two dashes in front (- – repoid). Unfortunately wordpress makes it one dash. Please, do not blame me 🙂

1. Create OEL.repo file in /etc/yum.repos.d and add the following entry

# vi /etc/yum.repos.d/OEL.repo

[oraclerpms]
name=OracleEnterpriseLinux
baseurl=http://yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
enabled=1

2. Install yum-utils and createrepo packages

# yum install yum-utils createrepo

3. It is better to add another disk and create repository there, do not use / disk(disk where OS is installed) for repository. It is just a best practice.

Create necessary folders:

# mkdir -p /OEL_repo/OEL6/latest
# mkdir -p /OEL_repo/logs
# mkdir -p /OEL_repo/scripts

4. Download rpms.
If you are using proxy you may need to set proxy parameters like  export http_proxy=http://proxy_server_ip:proxy_port/

/usr/bin/reposync –newest-only –repoid=oraclerpms -p /OEL_repo/OEL6/latest

5. After download completes create repository on downloaded rpms:

/usr/bin/createrepo /OEL_repo/OEL6/latest/oraclerpms/getPackage/

####################
The above steps should be done every time you want to update your local repository, so let’s create script and place all them in one file.

vi /OEL_repo/scripts/repo_sync.sh

#!/bin/bash

LOG_FILE=/OEL_repo/logs/repo_sync_$(date +%Y.%m.%d).log

# Delete old logs
find /OEL_repo/logs/repo_sync* -mtime +5 -delete; >> $LOG_FILE 2>&1

#Clean cache
yum clean all

# Sync repositories
/usr/bin/reposync /usr/bin/reposync –newest-only –repoid=oraclerpms -p /OEL_repo/OEL6/latest >> $LOG_FILE 2>&1

/usr/bin/createrepo /OEL_repo/OEL6/latest/oraclerpms/getPackage/ >> $LOG_FILE 2>&1

2. Change permissions for the executable script:

# chmod u+x /OEL_repo/scripts/repo_sync.sh

3. Set up a CRON job to run the script on a daily basis. The following entry runs the script each day at 01:00.

0 1 * * * /OEL_repo/scripts/repo_sync.sh > /dev/null 2>&1

###############

1. Install the Apache HTTP servers, start it and make sure it restarts automatically on reboot.

# yum install httpd
# service httpd start
# chkconfig httpd on

2. Create directory and place link there

mkdir -p /var/www/html/OEL_repo/OEL6/latest
ln -s /OEL_repo/OEL6/latest/oraclerpms/getPackage/ /var/www/html/OEL_repo/OEL6/latest/x86_64

# cd /var/www/html
# wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Note: /var/www/html is the default DocumentRoot directory. To check yours(sysadmin may have changed it) please check /etc/httpd/conf/httpd.conf file and find the keyword DocumentRoot

##############

1. Configure clients

vi /etc/yum.repos.d/local-ol6.repo

[oraclerpms]
name=Oracle Linux $releasever Latest ($basearch)
baseurl=http://RepositoryServerHostname.domain.ge/OEL_repo/OEL6/latest/$basearch/
gpgkey=http://RepositoryServerHostname.domain.ge/RPM-GPG-KEY-oracle-ol6
gpgcheck=1
enabled=1

Now you are able to install packages on clients by

# yum install ntp

Oracle: Audit DMLs by specific user

Ordinary auditing do not have option to indicate audit some activities done by specific user.

I mean, you cannot write the following:

audit insert on my_schema.my_table by my_user;  <<—-not possible. The right statement is:
audit insert on my_schema.my_table by access;
or
audit insert on my_schema.my_table by session;

If I want to audit only activities done by my_user, one of the way is to create audit policy like the following;

begin
dbms_fga.add_policy(
object_schema=>’my_schema‘,
object_name=> ‘my_table‘,
policy_name=> ‘my_policy’,
audit_condition => ‘sys_context(”USERENV”,”CURRENT_USER”)=”MY_USER”’,
enable => TRUE,
statement_types => ‘INSERT, UPDATE, DELETE’,
audit_column_opts => dbms_fga.all_columns);
END;

So audit_condition gives the opportunity to check something and in this case we are checking user that is running statements indicated in statement_types option.

–Logs will be located here

SELECT * FROM dba_fga_audit_trail

–To see what policies we have

SELECT * FROM dba_audit_policies