Creating a Snapshot-Based Backup of Oracle Linux Database Server(Exadata)

July 29, 2016 Leave a comment

Story: We have Exadata x5-2 servers and they must be PCI compliant. Our PCI scanner found a lot of vulnerabilities on these machines, most of them were rpm upgrades.

So we are in a hell (devil) . Exadata is not a toy and upgrading it’s system is not like playing :):)

We must backup the system , and take the image before doing such things.

Because of Exadata has LVM-s installed we can use LVM based backups.

So let’s start:

Creating a Snapshot-Based Backup of Oracle Linux Database Server with Uncustomized Partitions

This means that you have not changed anything after the initial installation.

a) create mount point where you save backups. It is better if you use NFS. But if you don’t have NFS you may temporarily use local storage and then copy backup to another disk to be more safe.

Exadata local disk is 1.6TB in size and there is about 1.4TB free , so we can create another logical volume for backup. Let’s create logical volume 400GB  in size.
Here, I can tell you the difference between DBA and Sysadmin 🙂 DBA needs more and more space and always takes more than required for being calm for next 5 years. And sysadmin is giving to DBA less space than required and everytime you ask him/her storage his/her face becomes like thinking_80_anim_gif

# lvcreate -L 400GB -n backup VGExaDb
# mkdir /backup
# mkfs.ext4 /dev/VGExaDb/backup
# echo “/dev/VGExaDb/backup     /backup                 ext4    defaults        0 0” >> /etc/fstab
# mount /backup

b) Take a snapshot-based backup of the / (root), /u01. Name them root_snap and u01_snap. Mount them to the newly created directories called exadata_os_backup and exadata_u01_backup.

# mkdir /backup/exadata_os_backup
# mkdir /backup/exadata_u01_backup
# lvcreate -L1G -s -c 32K -n root_snap /dev/VGExaDb/LVDbSys1
Logical volume “root_snap” created.
# e2label /dev/VGExaDb/root_snap DBSYS_SNAP
# mount /dev/VGExaDb/root_snap /backup/exadata_os_backup -t ext4
# lvcreate -L5G -s -c 32K -n u01_snap /dev/VGExaDb/LVDbOra1
Logical volume “u01_snap” created.
# e2label /dev/VGExaDb/u01_snap DBORA_SNAP
# mount /dev/VGExaDb/u01_snap  /backup/exadata_u01_backup -t ext4

c)  Create the backup file using the following command:

# cd  /backup
# tar -pjcvf /backup/os_backup.tar.bz2 * –exclude os_backup.tar.bz2 > /backup/os_backup.stdout 2> /backup/os_backup.stderr

Check the /tmp/backup_tar.stderr file for any significant errors. Errors about failing to tar open sockets, and other similar errors, can be ignored.

d) Unmount the snapshots and remove the snapshots for the root and /01 directories using the following commands:

umount /backup/exadata_u01_backup
umount /backup/exadata_os_backup
lvremove /dev/VGExaDb/u01_snap
lvremove /dev/VGExaDb/root_snap

More information:
https://docs.oracle.com/cd/E50790_01/doc/doc.121/e51951/db_server.htm#DBMMN21380
There is also discussed “Creating a Snapshot-Based Backup of Oracle Linux Database Server with Customized Partitions”.

 

Filed under Linux, Real World Scenarios Tagged with , , , , ,

Create a Local Yum Repository for Oracle Linux 6

July 12, 2016 Leave a comment

0. Just to note –repoid parameter has two dashes in front (- – repoid). Unfortunately wordpress makes it one dash. Please, do not blame me 🙂

1. Create OEL.repo file in /etc/yum.repos.d and add the following entry

# vi /etc/yum.repos.d/OEL.repo

[oraclerpms]
name=OracleEnterpriseLinux
baseurl=http://yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
enabled=1

2. Install yum-utils and createrepo packages

# yum install yum-utils createrepo

3. It is better to add another disk and create repository there, do not use / disk(disk where OS is installed) for repository. It is just a best practice.

Create necessary folders:

# mkdir -p /OEL_repo/OEL6/latest
# mkdir -p /OEL_repo/logs
# mkdir -p /OEL_repo/scripts

4. Download rpms.
If you are using proxy you may need to set proxy parameters like  export http_proxy=http://proxy_server_ip:proxy_port/

/usr/bin/reposync –newest-only –repoid=oraclerpms -p /OEL_repo/OEL6/latest

5. After download completes create repository on downloaded rpms:

/usr/bin/createrepo /OEL_repo/OEL6/latest/oraclerpms/getPackage/

####################
The above steps should be done every time you want to update your local repository, so let’s create script and place all them in one file.

vi /OEL_repo/scripts/repo_sync.sh

#!/bin/bash

LOG_FILE=/OEL_repo/logs/repo_sync_$(date +%Y.%m.%d).log

# Delete old logs
find /OEL_repo/logs/repo_sync* -mtime +5 -delete; >> $LOG_FILE 2>&1

#Clean cache
yum clean all

# Sync repositories
/usr/bin/reposync /usr/bin/reposync –newest-only –repoid=oraclerpms -p /OEL_repo/OEL6/latest >> $LOG_FILE 2>&1

/usr/bin/createrepo /OEL_repo/OEL6/latest/oraclerpms/getPackage/ >> $LOG_FILE 2>&1

2. Change permissions for the executable script:

# chmod u+x /OEL_repo/scripts/repo_sync.sh

3. Set up a CRON job to run the script on a daily basis. The following entry runs the script each day at 01:00.

0 1 * * * /OEL_repo/scripts/repo_sync.sh > /dev/null 2>&1

###############

1. Install the Apache HTTP servers, start it and make sure it restarts automatically on reboot.

# yum install httpd
# service httpd start
# chkconfig httpd on

2. Create directory and place link there

mkdir -p /var/www/html/OEL_repo/OEL6/latest
ln -s /OEL_repo/OEL6/latest/oraclerpms/getPackage/ /var/www/html/OEL_repo/OEL6/latest/x86_64

# cd /var/www/html
# wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Note: /var/www/html is the default DocumentRoot directory. To check yours(sysadmin may have changed it) please check /etc/httpd/conf/httpd.conf file and find the keyword DocumentRoot

##############

1. Configure clients

vi /etc/yum.repos.d/local-ol6.repo

[oraclerpms]
name=Oracle Linux $releasever Latest ($basearch)
baseurl=http://RepositoryServerHostname.domain.ge/OEL_repo/OEL6/latest/$basearch/
gpgkey=http://RepositoryServerHostname.domain.ge/RPM-GPG-KEY-oracle-ol6
gpgcheck=1
enabled=1

Now you are able to install packages on clients by

# yum install ntp

Filed under Linux, Real World Scenarios Tagged with , , , , ,

ASM Diskstring default values

July 7, 2016 Leave a comment

In case you have trouble getting the newly added disks to your ASM instance check for the ASM_DISKSTRING init paramter. If ASM_DISKSTRING is NOT SET then the following default is used

Default ASM_DISKSTRING per Operating System

Operating System Default            Search String
=======================================
Solaris (32/64 bit)                        /dev/rdsk/*
Windows NT/XP                          .orcldisk*
Linux (32/64 bit)                          /dev/raw/*

LINUX (ASMLIB)                         ORCL:*
LINUX (ASMLIB)                        /dev/oracleasm/disks/*

HPUX                                             /dev/rdsk/*
HP-UX(Tru 64)                            /dev/rdisk/*
AIX                                                 /dev/*

IF ASM_DISKSTRING is SET  then we should verify that the setting includes the disks that are needed to be seen by ASM.

 

Source from :  https://abchatur.wordpress.com/2011/09/01/asm-diskstring-default-values/

 

Filed under ASM, DB Tagged with , ,

Find database bit version (64bit, 32bit)

July 6, 2016 Leave a comment

There are several ways to determine that.  But let’s discuss just one version for Linux, one for Windows and one for Database.  🙂

If OS is Linux you can check it by file command.

 file $ORACLE_HOME/bin/oracle

/u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle: setuid setgid ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped

So my oracle is 64bit.

If it is windows:

Go to %ORACLE_HOME%\inventory\ContentsXML folder and open comps.xml file

Look for <DEP_LIST>
If following lines have

  • PLAT="NT_AMD64" then this Oracle Home is 64 bit.
  • PLAT="NT_X86" then – 32 bit.

The above commands was from OSs itself. You can get this information from database also.

select metadata
from sys.kopm$ ;

0000006001240F050B0C030C0C0504050D0609070805050505050F0
5050505050A050505050504050607080823472347081123081141B047…

If you see B047 then software is 64bit and if there is B023 then 32bit.

 

 

Filed under DB Tagged with , ,

Install Oracle 12.1.0.1 Error in invoking target ‘links proc gen_pcscfg procob’ of makefile …ins_precomp.mk

July 5, 2016 6 Comments

If you are installing 12.1.0.1 there are some problems. Oracle 12c (12.1.0.2) installs cleanly.

 

During the linking phase, you will see the following error.

Error in invoking target 'links proc gen_pcscfg procob' of makefile
'/u01/app/oracle/product/12.1.0.2/db_1/precomp/lib/ins_precomp.mk'.
See
'/u01/app/oraInventory/logs/installActions2014-04-26_08-07-04PM.log'
for details.

To fix it, do the following:

rm -rf $ORACLE_HOME/lib/stubs/*
cp $ORACLE_HOME/rdbms/lib/env_rdbms.mk $ORACLE_HOME/rdbms/lib/env_rdbms.mk.orig

Perform the following modifications to the “$ORACLE_HOME/rdbms/lib/env_rdbms.mk” file.

# Line 176
# FROM:
LINKTTLIBS=$(LLIBCLNTSH) $(ORACLETTLIBS) $(LINKLDLIBS)
# TO  :
LINKTTLIBS=$(LLIBCLNTSH) $(ORACLETTLIBS) $(LINKLDLIBS) -lons

# Line 279-280
# FROM:
LINK=$(FORT_CMD) $(PURECMDS) $(ORALD) $(LDFLAGS) $(COMPSOBJS)
LINK32=$(FORT_CMD) $(PURECMDS) $(ORALD) $(LDFLAGS32) $(COMPSOBJS)
# TO  :
LINK=$(FORT_CMD) $(PURECMDS) $(ORALD) $(LDFLAGS) $(COMPSOBJS) -Wl,--no-as-needed
LINK32=$(FORT_CMD) $(PURECMDS) $(ORALD) $(LDFLAGS32) $(COMPSOBJS) -Wl,--no-as-needed

# Line 3041-3042
# FROM:
TG4PWD_LINKLINE= $(LINK) $(OPT) $(TG4PWDMAI) \
        $(LLIBTHREAD) $(LLIBCLNTSH) $(LINKLDLIBS)
# TO  :
TG4PWD_LINKLINE= $(LINK) $(OPT) $(TG4PWDMAI) \
        $(LLIBTHREAD) $(LLIBCLNTSH) $(LINKLDLIBS) -lnnz12

Click the “Retry” button.

 

Filed under 12c, DB, Real World Scenarios Tagged with , ,

Oracle Clusterware is restarting database “” | shut down instance “” of database “” | start up instance “” of database “”

July 4, 2016 Leave a comment

When I used data guard broker to switchover primary database to standby database sometimes broker writes “Oracle Clusterware is restarting database”  … hangs and times out.

The problem was that database was not registered with srvctl.

srvctl add database -d db_unique_name -o oracle_home
srvctl add instance -d db_unique_name -i instance_name1 -n node_name1
srvctl add instance -d db_unique_name -i instance_name2 -n node_name2

instead of db_unique_name, oracle_home , instance_name1, node_name1,  instance_name2, node_name2 enter values according to your database. 

Filed under 12c, Data Guard Broker, DB, RAC, STANDBY Tagged with , , , , ,