Upgrading/Installing TFA with OSWatcher

The whole process is very simple and straightforward.
Post seems big but most of the content is a command output.

1. Download TFA Collector – TFA with Database Support Tools Bundle from Doc ID 1513912.1

2. Place downloaded zip file on rac1 and unzip it:

# cd /u01/app/sw
# ll
…
-rw-r--r-- 1 root root      264751391 Apr 25 19:18 TFA-LINUX_v19.2.1

# unzip TFA-LINUX_v19.2.1 

3. Install TFA:

[root@rac1 sw]# ./installTFA-LINUX 

TFA Installation Log will be written to File : /tmp/tfa_install_21556_2019_06_03-10_39_10.log
Starting TFA installation
 TFA Version: 192100 Build Date: 201904251105
 TFA HOME : /u01/app/12.2.0/grid/tfa/rac1/tfa_home
 Installed Build Version: 184100 Build Date: 201902262137
 TFA is already installed. Upgrading TFA
 TFA Upgrade Log : /u01/app/12.2.0/grid/tfa/rac1/tfapatch.log
 TFA will be upgraded on : 
 rac1
 rac2
 Do you want to continue with TFA Upgrade ? [Y|N] [Y]: Y
 Checking for ssh equivalency in rac2
 Node rac2 is not configured for ssh user equivalency
 SSH is not configured on these nodes : 
 rac2
 Do you want to configure SSH on these nodes ? [Y|N] [Y]: N
 Patching remote nodes using TFA Installer /u01/app/sw/installTFA-LINUX…
 Copying TFA Installer to rac2…
 lost connection
 Starting TFA Installer on rac2…
 Upgrading TFA on rac1 :
 Stopping TFA Support Tools…
 Shutting down TFA for Patching…
 Shutting down TFA
 Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.
 Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.
 Successfully shutdown TFA..
 No Berkeley DB upgrade required
 Copying TFA Certificates…
 Starting TFA in rac1…
 Starting TFA..
 Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Waiting up to 100 seconds for TFA to be started..
 . . . . . 
 Successfully started TFA Process..
 . . . . . 
 TFA Started and listening for commands
 Enabling Access for Non-root Users on rac1…
 Connection refused!rac2
 RemoteUtil : Connection refused!rac2
 .------------------------------------------------------------.
 | Host | TFA Version | TFA Build ID         | Upgrade Status |
 +------+-------------+----------------------+----------------+
 | rac1 |  19.2.1.0.0 | 19210020190425110550 | UPGRADED       |
 | rac2 | -           | -                    | NOT UPGRADED   |
 '------+-------------+----------------------+----------------'

The reason why it did not upgrade on rac2, is that I did not have ssh equivalency between nodes for root user.

I could enable ssh passwordless authentication and TFA would be upgraded in one step, but because of the security I will not enable it and just manually install TFA on the second node:

4. Copy installation file to rac2 and install:

[root@rac2 ~]# mkdir -p /u01/app/sw/
[root@rac2 ~]# chmod -R 777 /u01/app/sw/
[root@rac2 ~]# su - oracle
[oracle@rac2 ~]$ cd /u01/app/sw/  
[oracle@rac2 sw]$ scp rac1:/u01/app/sw/installTFA-LINUX .
installTFA-LINUX                           100%  254MB 114.9MB/s   00:02 
[root@rac2 sw]# ./installTFA-LINUX 
 TFA Installation Log will be written to File : /tmp/tfa_install_15370_2019_06_03-10_50_05.log
 Starting TFA installation
 TFA Version: 192100 Build Date: 201904251105
 TFA HOME : /u01/app/12.2.0/grid/tfa/rac2/tfa_home
 Installed Build Version: 184100 Build Date: 201902262137
 TFA is already installed. Upgrading TFA
 TFA Upgrade Log : /u01/app/12.2.0/grid/tfa/rac2/tfapatch.log
 TFA-00002 Oracle Trace File Analyzer (TFA) is not running
 TFA-00002 Oracle Trace File Analyzer (TFA) is not running
 Unable to determine the status of TFA in other nodes.
 TFA will be upgraded on Node rac2:
 Do you want to continue with TFA Upgrade ? [Y|N] [Y]: 
 Upgrading TFA on rac2 :
 Stopping TFA Support Tools…
 Shutting down TFA for Patching…
 Shutting down TFA
 Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.
 Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.
 . . . . . 
 . . . 
 Successfully shutdown TFA..
 No Berkeley DB upgrade required
 Copying TFA Certificates…
 Starting TFA in rac2…
 Starting TFA..
 Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
 Waiting up to 100 seconds for TFA to be started..
 . . . . . 
 Successfully started TFA Process..
 . . . . . 
 TFA Started and listening for commands
 Enabling Access for Non-root Users on rac2…
 .------------------------------------------------------------.
 | Host | TFA Version | TFA Build ID         | Upgrade Status |
 +------+-------------+----------------------+----------------+
 | rac2 |  19.2.1.0.0 | 19210020190425110550 | UPGRADED       |
 | rac1 |  19.2.1.0.0 | 19210020190425110550 | UPGRADED       |
 '------+-------------+----------------------+----------------'

5. Stop and Start TFA on rac1 and rac2:

# tfactl stop
Sending stoptfa
Success
Stopping TFA from the Command Line
Nothing to do !
LCM is not running
TFA is running  - Will wait 5 seconds (up to 3 times)  
TFA-00104 Cannot establish connection with TFA Server. Please check TFA Certificates
Killing TFA running with pid 16627
. . . 
Successfully stopped TFA..

# tfactl start
TFA-00002 Oracle Trace File Analyzer (TFA) is not running
Starting TFA..
Waiting up to 100 seconds for TFA to be started..
Successfully started TFA Process..

# ps -ef|grep OSWatcher
root     14528     1  0 May30 ?        00:44:11 /bin/sh ./OSWatcher.sh
root     14850 14528  0 May30 ?        00:00:58 /bin/sh ./OSWatcherFM.sh 48 /home/fg/oswbb/archive

6. Check OSWatcher repository location:

# ll /home/fg/oswbb/archive

drwxr-xr-x 2 root root  202 May 30 22:00 oswcpuinfo
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswifconfig
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswiostat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswmeminfo
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswmpstat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswnetstat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswnfsiostat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswpidstat
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswpidstatd
drwxr-xr-x 2 root root    6 May 30 21:56 oswprvtnet
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswps
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswslabinfo
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswtop
drwxr-xr-x 2 root root 4096 Jun  3 11:00 oswvmstat
drwxr-xr-x 2 root root    6 May 30 21:56 oswxentop

Linker error while installing Oracle 11g /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk

During installing 11g on OEL7 I got error when linking libraries.

To solve you should change syntax in the following file:

vi $ORACLE_HOME/sysman/lib/ins_emagent.mk

Change the following line from

$(MK_EMAGENT_NMECTL)

to:

$(MK_EMAGENT_NMECTL) -lnnz11

and click retry on error window to continue?

Good Luck!

Install Backup Exec Client(Remote Agent) on Linux

RALUS is a remote agent installation , simply Linux client of Backup exec that can be retrieved from Backup Exec installation DVD in LinuxUnixMac folder.

Some details:

Client Server IP: 10.0.1.100
Client Hostname: orcl_node
Domain: sa.ge
Media Server IP: 192.168.1.100

Create some installation directory and locate RALUS installation file to this folder.

1. Installation

# mkdir /0
# cd /0
# tar -xvf RALUS_RMALS_RAMS-4164.5.tar
# ./installralus

Step 1:

Enter the system names separated by spaces on which to install RALUS: (oracle-node1.tbilisi.gov.ge)10.0.1.100

Checking system communication:

Checking OS version on 10.0.1.100 ………….. Linux 2.6.18-194.el5
Checking system support for 10.0.1.100 … Linux 2.6.18-194.el5 supported by RALUS

Initial system check completed successfully.

Press Enter.

Step 2:

Press Enter.

Step 3:

If the output is:

Checking file system space ……………. required space is available

Then  press Enter

Step 4:

To display the Remote Agent as a selection in a media server’s backup selection tree, and to be able to specify a local network for use between the Remote Agent and a media server, enter the names or IP addresses of the media servers that you want the Remote Agent on ‘10.0.1.100’ to communicate with.

An IP Address: XXX.XXX.XXX.XXX
A Host Name: COMPUTERNAME

Enter a directory host:192.168.1.100

Do you want to add another name or address for this agent? [y, n] (n)

Step 5:

If the information is correct press Enter

Step 6:

In this step beoper group will be created if it doesn’t exist and root user will be added to this group.

Press Enter

Press Enter

Press Enter (choosing y option by default)

Press Enter(choosing n option by default)

Press Enter (choosing y option by default)

Press Enter

Step 7 :

Checking VRTSralus package ………………………… not installed

Press Enter

Step 8:

Installing VRTSralus 13.0.4164 on 10.0.1.100 …… done 1 of 1 steps

Press Enter

Step 9 :

Creating configuration files on 10.0.1.100  ………………… Done

Press Enter

RALUS installation is complete, now it’s time to configure it.

2. Configuration on Client

Step 1:

Run AgentConfig:

# /opt/VRTSralus/bin/AgentConfig

Symantec Backup Exec Remote Agent Utility
Choose one of the following options:
1. Configure database access
2. Configure Oracle instance information
3. Quit
Please enter your selection: 1

Configuring machine information
Choose one of the following options:
1. Add system credentials for Oracle operations
2. Edit system credentials used for Oracle operations
3. Remove system credentials used for Oracle operations
4. View system credentials used for Oracle operations
5. Quit
Please enter your selection: 1
     Enter a user name that has local system credentials: oracle
Enter the password:
Re-enter password:
Validating credentials…….
Do you want to use a custom port to connect to the media server during Oracle operations? (Y/N): N
     Commit Oracle operation settings to the configuration file? (Y/N): Y
     SUCCESS: Successfully added the entry to the configuration file.

Configuring machine information
Choose one of the following options:
1. Add system credentials for Oracle operations
2. Edit system credentials used for Oracle operations
3. Remove system credentials used for Oracle operations
4. View system credentials used for Oracle operations
5. Quit
Please enter your selection: 5

Symantec Backup Exec Remote Agent Utility
Choose one of the following options:
1. Configure database access
2. Configure Oracle instance information
3. Quit
Please enter your selection: 2

If this computer is a RAC node, you must perform additional steps for configuration before you continue. Refer to the readme for these additional steps.

Configuring the Oracle Agent
Choose one of the following options:
1. Add a new Oracle instance to protect
2. Edit an existing Oracle instance
3. Delete an existing Oracle instance
4. View Oracle instance entries that have been added in the Remote Agent Utility
5. Quit
Please enter your selection: 1
     Select an Oracle instance to configure
Entry 1. orcl
Enter the number 0 to go back
Enter your selection: 1
     Enter the Oracle database SYSDBA user name: SYS
Enter the Oracle database SYSDBA password:
Re-enter password:
Validating credentials…….
Enter the media server name or IP address:      The length of the entered data is greater than the maximum permitted length.
Enter the media server name or IP address: 192.168.1.100
     Do you use a recovery catalog? (Y/N):N
     Do you want to use a customized job template? (Y/N): N
     Commit Oracle operation settings to the configuration file? (Y/N): Y
     Created symbolic link for /opt/VRTSralus/bin/libobk.so at /u01/app/oracle/product/11.2.0/db_1/lib/libobk.so
SUCCESS: Successfully added the entry to the configuration file.

Configuring the Oracle Agent
Choose one of the following options:
1. Add a new Oracle instance to protect
2. Edit an existing Oracle instance
3. Delete an existing Oracle instance
4. View Oracle instance entries that have been added in the Remote Agent Utility
5. Quit
Please enter your selection: 5

Symantec Backup Exec Remote Agent Utility
Choose one of the following options:
1. Configure database access
2. Configure Oracle instance information
3. Quit
Please enter your selection: 3

Start agent:

# /etc/init.d/VRTSralus.init start
Starting Symantec Backup Exec Remote Agent ……
Starting Symantec Backup Exec Remote Agent:                              [  OK  ]

3. Configuration on Media Server

To make media server see your client do the following steps:

Step 1:

Run Start->-All Programs>Symantec Backup Exec->Backup Exec 2010

Step 2:

Tools->Options->Oracle –>click Modify List button->click New button and fill the following fields, by the following way:

Oracle or DB2 Server: 10.0.1.100
Logon Account  ->>>>click New button –>click New button and fill the following fields, by the following way:

User name: oracle
Password: enter oracle password for client server
Confirm Password: enter the password again
Account Name: oracle

click OK.

You will also need to configure SYS  user account.

And choose oracle in Logon Account  field.

click OK.

Do the same steps for 2 more times, just change Oracle or DB2 Server filed to orcl_node(instead of 10.0.1.100) for the first time and orcl_node.sa.ge for the second time

Step 3:

On media server , in hosts file add client entry.

That is all.

Install Oracle 11.2.0.3 with ASM on Centos 6.3

For RHEL 6, Oracle will provide ASMLib software and updates only when configured with a kernel distributed by Oracle. Oracle will not provide ASMLib packages for kernels distributed by Red Hat as part of RHEL 6. ASMLib updates will be delivered via Unbreakable Linux Network (ULN), which is available to customers with Oracle Linux support. ULN works with both Oracle Linux or Red Hat Linux installations, but ASMLib usage will require replacing any Red Hat kernel with a kernel provided by Oracle.

Because of the above announcement we use UDEV rules to prepare disks for ASM installation.

So let’s start.

1. Install required RPMs.

RPM names:

compat-libcap1-1.10-1.i686.rpm
compat-libcap1-1.10-1.x86_64.rpm
compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm
elfutils-devel-0.152-1.el6.x86_64.rpm
elfutils-libelf-devel-0.152-1.el6.x86_64.rpm
gcc-c++-4.4.6-4.el6.x86_64.rpm
glibc-2.12-1.80.el6.i686.rpm
glibc-devel-2.12-1.80.el6.i686.rpm
libaio-0.3.107-10.el6.i686.rpm
libaio-devel-0.3.107-10.el6.x86_64.rpm
libattr-2.4.44-7.el6.i686.rpm
libcap-2.16-5.5.el6.i686.rpm
libgcc-4.4.6-4.el6.i686.rpm
libstdc++-devel-4.4.6-4.el6.x86_64.rpm
libtool-ltdl-2.2.6-15.5.el6.i686.rpm
ncurses-devel-5.7-3.20090208.el6.i686.rpm
ncurses-libs-5.7-3.20090208.el6.i686.rpm
nss-softokn-freebl-3.12.9-11.el6.i686.rpm
pdksh-5.2.14-30.x86_64.rpm
readline-6.0-4.el6.i686.rpm

If you have Centos installation disk, these RPMs should be locate there. Or you can download them from  http://rpm.pbone.net/

Note: if during installing pdksh-5.2.14-30.x86_64.rpm it says that package conflicts with ksh then you should erase ksh package and install pdksh, like this:

# rpm -qa | grep ksh
# rpm -e ksh-…
# rpm –ivh pdksh-5.2.14-30.x86_64.rpm

2. Configure Kernel:

# vi /etc/sysctl.conf

kernel.shmall = 2097152
kernel.shmmax = 982431744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

To make changes take effect:

# sysctl –p

Edit /etc/pam.d/login :

# vi /etc/pam.d/login

session required pam_limits.so

Edit /etc/security/limits.conf:

# vi /etc/security/limits.conf

oracle soft  nproc   2047
oracle hard  nproc   16384
oracle soft  nofile  1024
oracle hard  nofile  65536

grid   soft  nproc   2047
grid   hard  nproc   16384
grid   soft  nofile  1024
grid   hard  nofile  65536

Run the following to add lines in /etc/pam.d/login:

[root@orcl ~]# cat >> /etc/pam.d/login <<EOF
session required pam_limits.so
EOF

Run the following to add lines in /etc/profile:

[root@orcl ~]# cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ] || [ \$USER = "grid" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF

Run the following to add lines in /etc/csh.login

[root@orcl ~]# cat >> /etc/csh.login <<EOF
if ( \$USER == "oracle" || \$USER == "grid" )
then
limit maxproc 16384
limit descriptors 65536
endif
EOF

Disable SELinux:

# /usr/sbin/getenforce
Enforcing

# /usr/sbin/setenforce 0

To make changes permanent, change /etc/sysconfig/selinux file by the following way:

cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing – SELinux security policy is enforced.
#     permissive – SELinux prints warnings instead of enforcing.
#     disabled – No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted – Targeted processes are protected,
#     mls – Multi Level Security protection.
SELINUXTYPE=targeted

To check the status again:

# /usr/sbin/getenforce
Disabled

3.  Creating OS groups and users.

#Creating groups for Grid Infrastructure

groupadd asmadmin
groupadd asmdba
groupadd asmoper

#Creating groups for Oracle Software

groupadd oinstall
groupadd dba
groupadd oper

#Creating user for Grid Infrastructure

useradd -g oinstall -G dba,asmadmin,asmdba,asmoper -d /home/grid grid

#Creating user for Oracle Software

useradd -g oinstall -G dba,oper,asmdba -d /home/oracle oracle

#Setting password for users

passwd grid
passwd oracle

4. Creating necessary directories

mkdir -p /u01/app/grid
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01

5. Creating .bash_profile-s

#For Oracle user

su – oracle
vi .bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=orcl; export ORACLE_SID

ORACLE_UNQNAME=orcl; export ORACLE_UNQNAME

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export ORACLE_HOME

ORACLE_TERM=xterm; export ORACLE_TERM

NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"
export NLS_DATE_FORMAT

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

THREADS_FLAG=native; export THREADS_FLAG

export TEMP=/tmp
export TMPDIR=/tmp

umask 022

#For Grid user

su – grid
vi .bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=+ASM; export ORACLE_SID

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/grid; export ORACLE_BASE

ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME

ORACLE_TERM=xterm; export ORACLE_TERM

NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

THREADS_FLAG=native; export THREADS_FLAG

export TEMP=/tmp
export TMPDIR=/tmp

umask 022

 

6. Add disks for ASM

If your disk is not partitioned yet, partition it by fdisk utility.

At this point you will need SCSI identifier.

So if your disk is physical just run the following command to identify it:

# scsi_id -g -u -d /dev/sdb
36000c292dfddac7b8934d3293313098e

Or if you have virtual disk , you will need to set disk.EnableUUID parameter to TRUE to see this identifier:

Shutdown VM, go to the directory where VM files are stored and edit VMX file. Add the following line:

disk.EnableUUID = "TRUE"

Restart your VM and run the above command (scsi_id -g -u -d /dev/sdb)

We will use this identifier for the UDEV rules to set permissions and alias for the new device in /etc/udev/rules.d/50-udev.rules file.

# vi /etc/udev/rules.d/50-udev.rules

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",
RESULT=="36000c292dfddac7b8934d3293313098e", NAME="oracleasm/asm-disk1",
OWNER="oracle", GROUP="dba", MODE="0660"

Restart udev, by the following way:

# /sbin/start_udev
Starting udev:                                             [  OK  ]

Check that alias exists:

# ls -la /dev/oracleasm/*
brw-rw—- 1 oracle dba 8, 17 Feb 18 12:49 /dev/oracleasm/asm-disk1

7. Install grid

Go to the grid installation folder and run:

./runInstaller

Step 1:

Skip Software Updates.

Step 2:

Configure Oracle Grid Infrastructure for a Standalone Server.

Step 3:

Click Next.

Step 4:

Select the Change Discovery Path button and enter /dev/oracleasm.

There should appear /dev/oracleasm/asm-disk-1 and check it.

Type disk group name as DATA01.

Step 5:

Set the passwords for the SYS and ASMSNMP accounts:

Step 6:

ASM Database Administrator(OSDBA) Group : asmdba
ASM Instance Administration Operator(OSOPER) Group: asmoper
ASM Instance Administrator(OSASM) Group: asmadmin

Step 7:

Click Next.

Step 8:

Click Next.

Step 9:

Click Install.

After pop-upping the window , asking to run

      • /u01/app/oraInventory/orainstRoot.sh
  • /u01/app/oracle/product/11.2.0/db_1/root.sh

run these scripts one by one and click OK on pop-up window.

Note: if root.sh script shows the error like this:

…error while loading shared libraries: libcap.so.1: …

Then in your system libcap-1 and libcap-2 RPMs are missing, first install them.

8. Install database

Go to the database installation folder and run:

./runInstaller

Step 1:

I don’t want any more spam thanks.

Step 2:

Skip the updates.

Step 3:

Create and configure a database.

Step 4:

Server Class.

Step 5:

Single instance database installation.

Step 6:

Advanced install.

Step 7:

Choose the languages you want.

Step 8:

Enterprise Edition.

Step 9:

Choose the defaults.
Note: Grid and Database must be in the different folders.

Step 10:

General Purpose / Transaction Processing.

Step 11:

Write a database name.

Step 12:

Use Oracle Enterprise Manager Database Control for database management.

Step 13:

Use Automatic Storage Management.

Step 15:

Do not enable automated backups

Step 16:

Select the DATA01 diskgroup.

Step 17:

Set the passwords for the database.

Step 18:

Accept the defaults.

Database Administrator(OSDBA) Group: dba
Database Operator(OSOPER) Group: oper

Step 19:

Click Install to start the installer.

After the installation requires to run root.sh script, run it.

That is all.

Installing standalone standby database for RAC

In this post we will install standalone standby database for 2-node RAC. Assume that we have already configured RAC, if not see the instructions here.

RAC description:

Instance name: orcl1
Hostname        : node1

Instance name: orcl2
Hostname        : node2

Standby description:

Instance name will be: stbydb
Hostname                    :  stbynode

All of these instances are using ASM.

So let’s start…

On primary database:

0. Add entries in hosts file.

On standby :

127.0.0.1 localhost

192.168.34.150 node1
192.168.34.151 node2

10.10.2.50 node1-priv
10.10.2.51 node2-priv

192.168.34.154 node1-vip
192.168.34.155 node2-vip

192.168.34.160 rac-scan

192.168.34.156 stbynode

On node1,node2:

127.0.0.1               localhost.localdomain localhost

192.168.34.150 node1
192.168.34.151 node2

10.10.2.50 node1-priv
10.10.2.51 node2-priv

192.168.34.154 node1-vip
192.168.34.155 node2-vip

192.168.34.160 rac-scan

192.168.34.156 stbynode

1. Enable archiving.

On node1:

[oracle@node1 ~]$ export ORACLE_SID=orcl1

sqlplus / as sysdba

SQL> select log_mode from v$database;

LOG_MODE
————
NOARCHIVELOG

SQL> shutdown immediate;

SQL> startup mount;

On node2:

export ORACLE_SID=orcl2

sqlplus / as sysdba

SQL> shutdown immediate;

SQL> startup mount;

Note that: In RAC environment to take database to archivelog mode , you must take all instances into mount state.

On node1:

SQL> alter database archivelog;

SQL> alter database open;

On node2:

SQL> alter database open;

On any node:

SQL> select log_mode from v$database;

LOG_MODE
————
ARCHIVELOG

2. Enable force logging.

On node1:

SQL> select force_logging from v$database;

FOR

NO

SQL> alter database force logging;

SQL> select force_logging from v$database;

FOR

YES

3. Modify the following parameters:

On node1:

SQL> show parameter pfile

NAME                                 TYPE        VALUE
———————————— ———– ——————————
spfile                               string      +DATA/orcl/spfileorcl.ora

SQL> create pfile from spfile;

Edit the following parameters into initorcl1.ora:

log_file_name_convert=’+DATA/stbydb’,’+DATA/orcl’
db_file_name_convert=’+DATA/stbydb’,’+DATA/orcl’
fal_client=’orcl1′
fal_client=’orcl2′
fal_server=’stbydb’
log_archive_config=’dg_config=(orcl,stbydb)’
log_archive_dest_state_1=’enable’
log_archive_dest_state_2=’ENABLE’
log_archive_dest_state_3=’enable’
log_archive_dest_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST  VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=orcl’
log_archive_dest_2=’SERVICE=stbydb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stbydb’
log_archive_dest_3=’LOCATION=+DATA/orcl/STANDBYLOG  VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=orcl’
log_archive_max_processes=7
log_archive_min_succeed_dest=2
remote_login_passwordfile=’EXCLUSIVE’
standby_file_management=’auto’
sec_case_sensitive_logon=FALSE

My initorcl1.ora looks like this:

orcl1.__db_cache_size=503316480
orcl2.__db_cache_size=486539264
orcl1.__java_pool_size=16777216
orcl2.__java_pool_size=16777216
orcl1.__large_pool_size=16777216
orcl2.__large_pool_size=16777216
orcl1.__oracle_base=’/u01/app/oracle’#ORACLE_BASE set from environment
orcl2.__oracle_base=’/u01/app/oracle’#ORACLE_BASE set from environment
orcl1.__pga_aggregate_target=587202560
orcl2.__pga_aggregate_target=587202560
orcl1.__sga_target=872415232
orcl2.__sga_target=872415232
orcl1.__shared_io_pool_size=0
orcl2.__shared_io_pool_size=0
orcl1.__shared_pool_size=318767104
orcl2.__shared_pool_size=335544320
orcl1.__streams_pool_size=0
orcl2.__streams_pool_size=0
*.audit_file_dest=’/u01/app/oracle/admin/orcl/adump’
*.audit_trail=’db’
*.cluster_database=true
*.compatible=’11.2.0.0.0′
*.control_files=’+DATA/orcl/controlfile/current.260.793127259′
*.db_block_size=8192
*.db_create_file_dest=’+DATA’
*.db_domain=”
*.db_name=’orcl’
*.db_recovery_file_dest=’+DATA’
*.db_recovery_file_dest_size=4039114752
*.diagnostic_dest=’/u01/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=orclXDB)’
orcl2.instance_number=2
orcl1.instance_number=1
*.log_archive_format=’%t_%s_%r.dbf’
*.memory_target=1449132032
*.open_cursors=300
*.processes=150
*.remote_listener=’rac-scan:1521′
*.remote_login_passwordfile=’exclusive’
orcl1.thread=1
orcl2.thread=2
orcl2.undo_tablespace=’UNDOTBS1′
orcl1.undo_tablespace=’UNDOTBS2′
log_file_name_convert=’+DATA/stbydb’,’+DATA/orcl’
db_file_name_convert=’+DATA/stbydb’,’+DATA/orcl’
fal_client=’orcl1′
fal_client=’orcl2′
fal_server=’stbydb’
log_archive_config=’dg_config=(orcl,stbydb)’
log_archive_dest_state_1=’enable’
log_archive_dest_state_2=’ENABLE’
log_archive_dest_state_3=’enable’
log_archive_dest_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST  VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=orcl’
log_archive_dest_2=’SERVICE=stbydb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stbydb’
log_archive_dest_3=’LOCATION=+DATA/orcl/STANDBYLOG  VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=orcl’
log_archive_max_processes=7
log_archive_min_succeed_dest=2
remote_login_passwordfile=’EXCLUSIVE’
standby_file_management=’auto’
*.sec_case_sensitive_logon=FALSE

SQL> shutdown immediate;

SQL> startup nomount pfile=’/u01/app/oracle/product/11.2.0/db_1/dbs/initorcl1.ora’

On node2:

[oracle@node2 ~]$ export ORACLE_SID=orcl2
[oracle@node2 ~]$ sqlplus / as sysdba

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

On node1:

SQL> create spfile=’+DATA/ORCL/spfileorcl.ora’ from pfile=’/u01/app/oracle/product/11.2.0/db_1/dbs/initorcl1.ora’;

Edit initorcl1.ora file and write just the following entry:

SPFILE=’+DATA/ORCL/spfileorcl.ora’

And start the instance.

SQL> shutdown immediate;

SQL> startup

On node2:

If you try to open node2, it will stuck because it will try to send archivelog to standby location, which is not completely configured yet. So let node2 to be in down state.

3. Create same password files for all instances(passwords must be the same ).

On node1:

[oracle@node1 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapworcl1 password=oracle force=y entries=5 ignorecase=y

On node2:

[oracle@node2 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapworcl2 password=oracle force=y entries=5 ignorecase=y

On stbynode:

[oracle@stbynode ~]$ orapwd file=$ORACLE_HOME/dbs/orapwstbydb password=oracle entries=5 force=y  ignorecase=y

4. Create standby logfiles on primary.

On node1:

SQL> sqlplus / as sysdba
SQL>  select max(length(member)) from v$logfile;

MAX(LENGTH(MEMBER))
——————-
42

SQL> col member for a42
SQL> select group#,member,type from v$logfile order by 1;

GROUP# MEMBER                                     TYPE
———- —————————————— ——-
1 +DATA/orcl/onlinelog/group_1.261.793127267 ONLINE
2 +DATA/orcl/onlinelog/group_2.262.793127267 ONLINE
3 +DATA/orcl/onlinelog/group_3.266.793128605 ONLINE
4 +DATA/orcl/onlinelog/group_4.267.793128605 ONLINE

SQL> alter database add standby logfile thread 1;

SQL> alter database add standby logfile thread 2;

SQL> select group#,member,type from v$logfile order by 1;

GROUP# MEMBER                                     TYPE
———- —————————————— ——-
1 +DATA/orcl/onlinelog/group_1.261.793127267 ONLINE
2 +DATA/orcl/onlinelog/group_2.262.793127267 ONLINE
3 +DATA/orcl/onlinelog/group_3.266.793128605 ONLINE
4 +DATA/orcl/onlinelog/group_4.267.793128605 ONLINE
5 +DATA/orcl/onlinelog/group_5.273.793983817 STANDBY
5 +DATA/orcl/onlinelog/group_5.274.793983819 STANDBY
6 +DATA/orcl/onlinelog/group_6.271.793983823 STANDBY
6 +DATA/orcl/onlinelog/group_6.272.793983823 STANDBY

5. Create parameter file for standby database.

SQL> create pfile=’$ORACLE_HOME/dbs/initstbydb.ora’ from spfile;

Edit required parameters. Delete red entries. Green fields are corrected entries:

orcl2.__db_cache_size=486539264
orcl1.__
db_cache_size=503316480
orcl2.__java_pool_size=16777216
orcl1.__
java_pool_size=16777216
orcl2.__large_pool_size=16777216
orcl1.__
large_pool_size=16777216
orcl1.__oracle_base=’/u01/app/oracle’#ORACLE_BASE set from environment
orcl2.__oracle_base=’/u01/app/oracle’#ORACLE_BASE set from environment
orcl2.__pga_aggregate_target=587202560
orcl1.__
pga_aggregate_target=587202560
orcl2.__sga_target=872415232
orcl1.__
sga_target=872415232
orcl2.__shared_io_pool_size=0
orcl1.__shared_io_pool_size=0
orcl2.__shared_pool_size=335544320
orcl1.__
shared_pool_size=318767104
orcl2.__streams_pool_size=0
orcl1.__
streams_pool_size=0
*.audit_file_dest=’/u01/app/oracle/admin/stbydb/adump’
*.audit_trail=’db’
*.cluster_database=true
*.compatible=’11.2.0.0.0′
*.control_files=’+DATA/stbydb/controlfile/control01.ctl’
*.db_block_size=8192
*.db_create_file_dest=’+DATA’
*.db_domain=”
*.db_file_name_convert=‘+DATA/orcl’,’+DATA/stbydb’
*.db_name=’orcl’
*.db_unique_name=’stbydb’
*.db_recovery_file_dest=’+DATA’
*.db_recovery_file_dest_size=4039114752
*.diagnostic_dest=’/u01/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=stbydbXDB)’
*.fal_client=’stbydb
*.fal_server=’orcl
orcl2.instance_number=2
orcl1.instance_number=1
*.log_archive_config=’dg_config=(orcl,stbydb)’
*.log_archive_dest_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST  VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=stbydb
*.log_archive_dest_2=’SERVICE=orcl LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl
*.log_archive_dest_3=’LOCATION=+DATA/stbydb/STANDBYLOG  VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=stbydb
*.log_archive_dest_state_1=’enable’
*.log_archive_dest_state_2=’ENABLE’
*.log_archive_dest_state_3=’enable’
*.log_archive_format=’%t_%s_%r.dbf’
*.log_archive_max_processes=7
*.log_archive_min_succeed_dest=2
*.log_file_name_convert=’+DATA/orcl‘,’+DATA/stbydb
*.memory_target=1449132032
*.open_cursors=300
*.processes=150
*.remote_listener=’rac-scan:1521′
*.remote_login_passwordfile=’EXCLUSIVE’
*.standby_file_management=’auto’
orcl1.thread=1
orcl2.thread=2
orcl2.
undo_tablespace=’UNDOTBS1′
orcl1.undo_tablespace=’UNDOTBS2′
sec_case_sensitive_logon=FALSE

—initstbydb.ora

db_cache_size=503316480
java_pool_size=16777216
large_pool_size=16777216
pga_aggregate_target=587202560
sga_target=872415232
shared_pool_size=318767104
streams_pool_size=0
*.audit_file_dest=’/u01/app/oracle/admin/stbydb/adump’
*.audit_trail=’db’
*.compatible=’11.2.0.0.0′
*.control_files=’+DATA/stbydb/controlfile/control01.ctl’
*.db_block_size=8192
*.db_create_file_dest=’+DATA’
*.db_domain=”
*.db_file_name_convert=’+DATA/orcl’,’+DATA/stbydb’
*.db_name=’orcl’
*.db_unique_name=’stbydb’
*.db_recovery_file_dest=’+DATA’
*.db_recovery_file_dest_size=4039114752
*.diagnostic_dest=’/u01/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=stbydbXDB)’
*.fal_client=’stbydb’
*.fal_server=’orcl’
*.log_archive_config=’dg_config=(stbydb,orcl)’
*.log_archive_dest_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST  VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=stbydb’
*.log_archive_dest_2=’SERVICE=orcl LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl’
*.log_archive_dest_3=’LOCATION=+DATA/stbydb/STANDBYLOG  VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=stbydb’
*.log_archive_dest_state_1=’enable’
*.log_archive_dest_state_2=’ENABLE’
*.log_archive_dest_state_3=’enable’
*.log_archive_format=’%t_%s_%r.dbf’
*.log_archive_max_processes=7
*.log_archive_min_succeed_dest=2
*.log_file_name_convert=’+DATA/orcl’,’+DATA/stbydb’
*.memory_target=1449132032
*.open_cursors=300
*.processes=150
*.remote_listener=’rac-scan:1521′
*.remote_login_passwordfile=’EXCLUSIVE’
STANDBY_FILE_MANAGEMENT=’auto’
sec_case_sensitive_logon=FALSE

Copy parameter file to standby.

[oracle@node1 dbs]$ scp initstbydb.ora oracle@stbynode:$ORACLE_HOME/dbs

The authenticity of host ‘stbynode (192.168.34.156)’ can’t be established.
RSA key fingerprint is 73:fa:1f:a0:04:41:68:98:30:25:28:55:59:9c:07:9b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘stbynode,192.168.34.156’ (RSA) to the list of known hosts.
oracle@stbynode’s password:
initstbydb.ora                                100% 1479     1.4KB/s   00:00

6. Create required directories on primary and standby databases.

On node1:

mkdir -p /u01/app/oracle/backup

On stbynode

mkdir -p  /u01/app/oracle/admin/stbydb/adump
mkdir -p /u01/app/oracle/backup

su – grid

[grid@stbynode ~]$ asmcmd
ASMCMD> ls
CRS/
DATA/
ASMCMD> cd data
ASMCMD> ls
ASMCMD> mkdir stbydb
ASMCMD> cd stbydb
ASMCMD> mkdir ARCHIVELOG CONTROLFILE DATAFILE ONLINELOG PARAMETERFILE STANDBYLOG TEMPFILE

7. Add tns entries in tnsnames.ora file and listener entry in listener.ora file:

–tnsnames.ora

On node1, node2 and stbynode:

ORCL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)

stbydb=
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = stbynode)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = stbydb)
)
)

On node1, node2 and stbynode:

[oracle@node2 dbs]$ tnsping orcl

TNS Ping Utility for Linux: Version 11.2.0.1.0 – Production on 14-SEP-2012 17:04:13

Copyright (c) 1997, 2009, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl)))
OK (20 msec)
[oracle@node2 dbs]$ tnsping stbydb

TNS Ping Utility for Linux: Version 11.2.0.1.0 – Production on 14-SEP-2012 17:04:20

Copyright (c) 1997, 2009, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = stbynode)(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = stbydb)))
OK (20 msec)

8. Add entry in oratab.

On stbynode:

vi /etc/oratab

+ASM:/u01/app/11.2.0/grid:N
stbydb:/u01/app/oracle/product/11.2.0/db_1:N

9. Making backups on primary(node1) and copy them to standby.

On node1:

export ORACLE_SID=orcl1
rman target /
RMAN> configure channel device type disk format ‘/u01/app/oracle/backup/%U’;
RMAN> backup database plus archivelog;
RMAN> backup current controlfile for standby;
RMAN> backup archivelog all;

[oracle@node1 ~]$ scp /u01/app/oracle/backup/* oracle@stbynode:/u01/app/oracle/backup/

10.  Duplicate database.

On stbynode:

[oracle@stbynode ~]$ export ORACLE_SID=stbydb
[oracle@stbynode ~]$ sqlplus / as sysdba

SQL> startup nomount;

[oracle@stbynode ~]$ export ORACLE_SID=stbydb
[oracle@stbynode ~]$ rman target sys/oracle@orcl auxiliary /

Recovery Manager: Release 11.2.0.1.0 – Production on Fri Sep 14 19:50:02 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1320907995)
connected to auxiliary database: ORCL (not mounted)

RMAN> duplicate target database for standby nofilenamecheck;

[oracle@stbynode ~]$ export ORACLE_SID=stbydb
[oracle@stbynode ~]$ sqlplus / as sysdba

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

11. Startup all.

On node2:

SQL> startup;

On stbynode:

alter database open;

12. Test.

On node1:

SQL> create table b(b number);
SQL> insert into b values(1);
SQL> commit;
SQL> alter system switch logfile;

On node2:

SQL> alter system switch logfile;

On stbynode:

SQL> select * from b;

B
———-
1

On node1:

ALTER SESSION SET nls_date_format=’DD-MON-YYYY HH24:MI:SS’;

SELECT sequence#, first_time, next_time
FROM v$archived_log
ORDER BY sequence#;

ALTER SYSTEM SWITCH LOGFILE;

SELECT sequence#, first_time, next_time
FROM v$archived_log
ORDER BY sequence#;

On node2:

SQL> alter system switch logfile;

On stbynode:

ALTER SESSION SET nls_date_format=’DD-MON-YYYY HH24:MI:SS’;

SELECT sequence#, first_time, next_time, applied
FROM v$archived_log
ORDER BY sequence#;

Install Oracle 11g RAC on Oracle Enterprise Linux 5

0. Brief description

We will install 2-node RAC for testing purposes. With OS Oracle Enterprise Linux 64bit. Database version 11g Release 2(11.2.0.1.0) 64bit. Grid version also will be 11hR2.

1. Pre-requisites

– Disk space:
— 30GB for root directory
— 2.3GB for database
— 2.95GB for Oracle
Grid Infrastructure
— approx. 4GB for swap

– Memory

     –4GB

– Installation Software

Enterprise Linux:https://edelivery.oracle.com/linux
Database: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html (Choose database)
Grid:http://www.oracle.com/technetwork/products/clusterware/downloads/index.html  (Choose Linux x86-64)

2. Installation

2.1 Configuring Servers

Create New Virtual Machine_Custom

Create New Virtual Machine_Name

Create New Virtual Machine_Datastore

Create New Virtual Machine_Version

Create New Virtual Machine_OS

Create New Virtual Machine_Processors
Create New Virtual Machine_Memory

Create New Virtual Machine_NIC

Create New Virtual Machine_Controller

Create New Virtual Machine_Create_New_VD
Create New Virtual Machine_Disk_Capacity

Create New Virtual Machine_Virtual_Device_node
Create New Virtual Machine_Settings_Summary

After creation edit virtual machine settings.  Specify Enterprise Linux

Create New Virtual Machine_CD

Click OK. Open the console. And power on the machine.

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup_Language

Virtual Virtual Machine_Enterprise_Linux_Setup_Keyboard

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup_partition

Virtual Machine_Enterprise_Linux_Setup_partition

Virtual Machine_Enterprise_Linux_Setup_partition

Note swap size should be chosen by the following way:

1. If you have RAM between 1 and 2G, swap should be 1.5 x RAM (1.5 times RAM) .
2. For RAM 2 to 8G , swap space has to be equal to RAM.
3. For RAM more than 8G, swap needs to be 3/4 RAM.

Because of my RAM is 4GB , I choose 4GB of swap space.

I left logical volumes because , increasing disk space with it is easier than with physical volumes.

Virtual Machine_Enterprise_Linux_Setup_bootloader

Virtual Machine_Enterprise_Linux_Setup_Network_Devices

Virtual Machine_Enterprise_Linux_Setup_Network_Devices

Virtual Machine_Enterprise_Linux_Setup_Network_Devices

Virtual Machine_Enterprise_Linux_Setup_Region

Virtual Machine_Enterprise_Linux_Setup_Root_Password

Virtual Machine_Enterprise_Linux_Setup

Desktop Environments ->GNOME Desktop Environment
Virtual Machine_Enterprise_Linux_Setup_Packages

Applications-> Editors, Graphical Internet, Text-based Internet

Virtual Machine_Enterprise_Linux_Setup_Packages

Developement->check all except KDE and Ruby

Virtual Machine_Enterprise_Linux_Setup_Packages

Click Development Tools and Optional packages button: check gcc44-4.4, gcc44-c++.

Click X Software Development and Optional packages button: check openmotif-devel.

Servers-> Server Configuration Tools

Virtual Machine_Enterprise_Linux_Setup_Packages

Base System-> Administration ToolsBase, Java, Legacy Software Support, System ToolsX Window System

Virtual Machine_Enterprise_Linux_Setup_Packages

Click Legacy Software Support and Optional packages button and choose: compat-db, openmotif22.

Click System Tools and Optional packages button and choose: 1:mc, oracleasm*, sysstat.

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup

Virtual Machine_Enterprise_Linux_Setup_License_Agreement

Virtual Machine_Enterprise_Linux_Setup_Firewall

Virtual Machine_Enterprise_Linux_Setup_SELinux

Virtual Machine_Enterprise_Linux_Setup_SELinux

Virtual Machine_Enterprise_Linux_Setup_Kdump

Virtual Machine_Enterprise_Linux_Setup_Date_and_Time

Virtual Machine_Enterprise_Linux_Setup_Create_user

Virtual Machine_Enterprise_Linux_Setup_Create_user

Virtual Machine_Enterprise_Linux_Setup_Sound_Card

Virtual Machine_Enterprise_Linux_Setup_Additional_CDs

Virtual Machine_Enterprise_Linux_Setup_Additional_CDs


2.2 Installing Guest additions

Virtual Machine_Enterprise_Linux_Setup_Guest_Additions

 

 

 

 

 

 

 

 

 

 

 

 

From command line:

mkdir /0
cp /media/VMware\ Tools/VMwareTools-8.3.2-257589.tar.gz /0
cd /0
tar -zxf VMwareTools-8.3.2-257589.tar.gz
cd vmware-tools-distrib/
./vmware-install.pl

After installation completes , open new terminal and run:
vmware-toolbox
Virtual Machine_Enterprise_Linux_Setup_vmwaretoolbox
2.2.1 Enabling copy/paste on vSphere
For the following step server should be  shutdowned:

“Starting with vSphere 4.1, the Copy and Paste options are, by default, disabled for security reasons.

To be able to copy and paste between the guest operating system and the remote console, you must enable the Copy and Paste options using the vSphere Client. Alternatively, you can use RDP (Remote Desktop Protocol) to connect to the Windows virtual machines.

To enable this option for a specific virtual machine:

1. Log into a vCenter Server system using the vSphere Client and power off the virtual machine.
2. Select the virtual machine and click the Summary tab.
3. Click Edit Settings.
4. Navigate to Options > Advanced > General and click Configuration Parameters.
5. Click Add Row.
6. Type these values in the Name and Value columns:
Name Value

      • isolation.tools.copy.disable    false
  • isolation.tools.paste.disable   false
    Note: These options override any settings made in the VMware Tools control panel of the guest operating system.

1. Click OK to close the Configuration Parameters dialog, and click OK again to close the Virtual Machine Properties dialog.
2. Power on the virtual machine.“ source is  from http://kb.vmware.com

2.3 Preparing Environment for Oracle Grid infrastructure

2.3.1 Install necessary RPM packages

For this we will need Oracle Enterprise Linux disk.

Virtual Machine_Enterprise_Linux_Setup_CD_DVD

[root@node1 ~]# cp -R /media/Enterprise\ Linux\ dvd\ 20100405/Server /0

Note: Server folder contains necessary rpms. Most of rpms may already been installed.

cd /0/Server
rpm -Uvh binutils-2.17.50.0.6-14.el5.x86_64.rpm
rpm -Uvh compat-libstdc++-33-3.2.3-61.x86_64.rpm
rpm -Uvh elfutils-libelf-devel-0.137-3.el5.x86_64.rpm
rpm -Uvh elfutils-libelf-devel-static-0.137-3.el5.x86_64.rpm
rpm -Uvh gcc-4.1.2-48.el5.x86_64.rpm
rpm -Uvh gcc-c++-4.1.2-48.el5.x86_64.rpm
rpm -Uvh glibc-devel-2.5-49.x86_64.rpm
rpm -Uvh glibc-headers-2.5-49.x86_64.rpm
rpm -Uvh kernel-headers-2.6.18-194.el5.x86_64.rpm
rpm -Uvh libacl-devel-2.2.39-6.el5.x86_64.rpm
rpm -Uvh libaio-devel-0.3.106-5.i386.rpm
rpm -Uvh libaio-0.3.106-5.x86_64.rpm
rpm -Uvh libaio-0.3.106-5.i386.rpm
rpm -Uvh libaio-devel-0.3.106-5.x86_64.rpm
rpm -Uvh libgomp-4.4.0-6.el5.x86_64.rpm
rpm -Uvh libstdc++-devel-4.1.2-48.el5.x86_64.rpm
rpm -Uvh libstdc++-devel-4.1.2-48.el5.x86_64.rpm
rpm -Uvh sysstat-7.0.2-3.el5.x86_64.rpm
rpm -Uvh unixODBC-2.2.11-7.1.x86_64.rpm
rpm -Uvh unixODBC-2.2.11-7.1.i386.rpm
rpm -Uvh unixODBC-devel-2.2.11-7.1.x86_64.rpm
rpm -Uvh unixODBC-devel-2.2.11-7.1.i386.rpm
rpm -Uvh ksh-20100202-1.el5.x86_64.rpm

2.3.3 Modify /etc/hosts file

vi /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost

#Public
192.168.34.150   node1
192.168.34.151   node2

#Private
10.10.2.50      node1-priv
10.10.2.51      node2-priv

#Virtual
192.168.34.154  node1-vip
192.168.34.155  node2-vip

#SCAN
192.168.34.160    rac-scan

SCAN should be defined on DNS. But for now we will use hosts file.

2.3.4 Modify kernel parameters in /etc/sysctl.conf

vi /etc/sysctl.conf

# Controls the maximum shared segment size, in bytes
#kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
#kernel.shmall = 4294967296

#Added by DBA
kernel.shmall                            = 2097152
kernel.shmmax                         = 2147483648
kernel.shmmni                          = 4096
kernel.sem                                 = 250 32000 100 128
fs.file-max                                   = 6815744
net.ipv4.ip_local_port_range  = 9000 65500
net.core.rmem_default            = 1048576
net.core.rmem_max                 = 4194304
net.core.wmem_default          = 262144
net.core.wmem_max              = 1048576
fs.aio-max-nr                            = 10448576

To make changes take effect :

/sbin/sysctl -p

2.3.5 Modify /etc/security/limits.conf

vi /etc/security/limits.conf

#*                        soft        core            0
#*                        hard      rss             10000
#@student        hard      nproc          20
#@faculty          soft        nproc          20
#@faculty          hard      nproc          50
#ftp                     hard      nproc           0
#@student        –            maxlogins   4

#Added By DBA
oracle           soft            nproc           2047
oracle           hard          nproc          16384
oracle           soft           nofile           1024
oracle           hard         nofile           65536

grid               soft            nproc           2047
grid               hard          nproc          16384
grid               soft            nofile           1024
grid               hard          nofile           65536


# End of file


2.3.6 Modify
/etc/pam.d/login

vi /etc/pam.d/login

#%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
auth       include      system-auth
account    required     pam_nologin.so
account    include      system-auth
password   include      system-auth

# pam_selinux.so close should be the first session rule
session    required     pam_selinux.so close
session    include      system-auth
session    required     pam_loginuid.so
session    optional     pam_console.so

# pam_selinux.so open should only be followed by sessions to be executed in the user context
session    required     pam_selinux.so open
session    optional     pam_keyinit.so force revoke
session required pam_limits.so

2.3.7 Deconfigure NTP

[root@node1 Server]# service ntpd stop
Shutting down ntpd:                                        [FAILED]
[root@node1 Server]# chkconfig ntpd off
[root@node1 Server]# mv /etc/ntp.conf /etc/ntp.conf.backup
[root@node1 Server]# rm /var/run/ntpd.pid
rm: cannot lstat `/var/run/ntpd.pid’: No such file or directory

2.3.8 Add necessary Groups and User, and create directories.

#Creating groups for Grid Infrastructure

groupadd asmadmin
groupadd asmdba
groupadd asmoper

#Creating groups for Oracle Software

groupadd oinstall
groupadd dba
groupadd oper

#Creating user for Grid Infrastructure

useradd -g oinstall -G dba,asmadmin,asmdba,asmoper -d /home/grid grid

#Creatnig user for Oracle Software

useradd -g oinstall -G dba,oper,asmdba -d /home/oracle oracle

#Setting password for users

passwd grid
passwd oracle

#Creating necessary directories

mkdir -p /u01/app/grid
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01

2.3.9 Edit .bash_profile

#For Oracle user

su – oracle
vi .bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=orcl1; export ORACLE_SID

ORACLE_UNQNAME=orcl1; export ORACLE_UNQNAME

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export ORACLE_HOME

ORACLE_TERM=xterm; export ORACLE_TERM

NLS_DATE_FORMAT=”DD-MON-YYYY HH24:MI:SS”
export NLS_DATE_FORMAT

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

THREADS_FLAG=native; export THREADS_FLAG

export TEMP=/tmp
export TMPDIR=/tmp

umask 022

#For Grid user

su – grid
vi .bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=+ASM1; export ORACLE_SID

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/grid; export ORACLE_BASE

ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME

ORACLE_TERM=xterm; export ORACLE_TERM

NLS_DATE_FORMAT=”DD-MON-YYYY HH24:MI:SS”; export NLS_DATE_FORMAT

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

THREADS_FLAG=native; export THREADS_FLAG

export TEMP=/tmp
export TMPDIR=/tmp

umask 022

2.3.10 Shutdown NODE1. Using clone, make NODE2 and configure it.

su –
[root@rac1 ~]# init 0

Clone Virtual Machine_Name

Clone Virtual Machine_Datastore

Clone Virtual Machine_Disk_Format

Clone Virtual Machine_Guest_Customization

Clone Virtual Machine_Ready_to_Complete

Open node2 console and run:

system-config-network

Virtual Machine_Network_Configuration

Delete .bak entries:

Virtual Machine_Network_Configuration

In DNS tab, change node1 to node2

Virtual Machine_Network_Configuration

In Devices tab, edit eth0 by the following way:

Virtual Machine_Network_Configuration

In the same window, in Hardware Device tab click Probe.

Virtual Machine_Network_Configuration

Click OK. Edit eth1.

Virtual Machine_Network_Configuration

In the same window, in Hardware Device tab click Probe.

Virtual Machine_Network_Configuration

Click OK. Activate eth0 and eth1 devices, Save(Ctrl+S) and Quite(Ctrl+Q)

Virtual Machine_Network_Configuration

Open terminal and run:

[root@node2 ~]# service network restart
Shutting down interface eth0:                               [  OK  ]
Shutting down interface eth1:                               [  OK  ]
Shutting down loopback interface:                      [  OK  ]
Bringing up loopback interface:                           [  OK  ]
Bringing up interface eth0:                                   [  OK  ]
Bringing up interface eth1:                                   [  OK  ]

Edit bash profiles:

#For oracle bash_profile change the following lines:

ORACLE_SID=orcl2; export ORACLE_SID
ORACLE_UNQNAME=orcl2; export ORACLE_UNQNAME

#For grid bash_profile change the following line:

ORACLE_SID=+ASM2; export ORACLE_SID

2.3.11  Add necessary shared disks.

Shutdown both nodes.

init 0

Edit virtual machine settings and add hardware:

Virtual Machine_Add_Hard_Disk

Virtual Virtual Machine_Add_Hard_Disk_New

Virtual Machine_Add_Hard_Disk_Capacity

Choose SCSI (1:0), different from SCSI specified for 30GB hard disk.

Virtual Machine_Add_Hard_Disk_SCSI

Virtual Machine_Add_Hard_Disk_Summary

Choose Virtual for New SCSI Controller:

Virtual Machine_SCSI_Controller_Virtual

Repeat the steps 2 more times. So create 2 more disks with 10GB space and 1 with 3GB for CRS.

The second hard disk should look like this. Note , choose SCSI(1:1).

Virtual Machine_Add_Hard_Disk_Summary

Virtual Machine_Add_Hard_Disk_Summary

Virtual Machine_Add_Hard_Disk_Summary

Now configure node2: Edit virtual machine settings and add hard disk

Virtual Machine_Add_Hard_Disk

Virtual Machine_Add_Hard_Disk_Existing

Virtual Machine_Add_Hard_Disk_Path

Virtual Machine_Add_Hard_Disk_SCSI

Virtual Machine_Add_Hard_Disk_Summary

Virtual Machine_Add_Hard_Disk_SCSI_Virtual

Do the same steps for other disks, just choose SCSI(1:1) for the first ,SCSI(1:2) for the second and SCSI(1:3) for the third  :

Virtual Machine_Add_Hard_Disk_Summary

Virtual Machine_Add_Hard_Disk_Summary

Virtual Machine_Add_Hard_Disk_Summary

Power on both machines.

Open terminal in node1 server and partition newly added disks.(You don’t need to do the same from node2 because, disks are shared and node2 will see partitions created by node1, but it needs to reboot)

[root@node1 ~]# cd /dev
[root@node1 dev]# ls sd*
sda  sda1  sda2  sdb  sdc  sdd  sde
[root@node1 dev]# fdisk sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):
Using default value 1305

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Do the same steps for 3 other disks.

Reboot node2.

2.3.12 Install and configure ASMLIB 

——————————-Download Link

http://www.oracle.com/technetwork/server-storage/linux/downloads/rhel5-084877.html

To know your kernel version:

[root@node1 dev]# uname -r
2.6.18-194.el5xen

I’ve downloaded just the following package, because others were already in Enterprise Linux installation disk.

oracleasmlib-2.0.4-1.el5.x86_64.rpm

——————————-Installation

To make visible oracleasmlib-2.0.4-1.el5.x86_64.rpm package for our guest OS , we should make .iso file from it and bound to the system as disk.  For this I use ISO Recorder 3.1

[root@node1 Server]# rpm -Uvh oracleasm-support-2.1.3-1.el5.x86_64.rpm
warning: oracleasm-support-2.1.3-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
package oracleasm-support-2.1.3-1.el5.x86_64 is already installed
[root@node1 Server]# rpm -Uvh oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
package oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64 is already installed
file /lib/modules/2.6.18-194.el5xen/kernel/drivers/addon/oracleasm/oracleasm.ko from install of oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64 conflicts with file from package oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64
[root@node1 Server]# rpm -Uvh oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
package oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.x86_64 is already installed
[root@node1 Server]# rpm -Uvh oracleasmlib-2.0.4-1.el5.x86_64.rpm
warning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:oracleasmlib           ########################################### [100%]

Do the same on node2.
—————————–Configuration

/usr/sbin/oracleasm configure –i

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

Do the same on node2.

# Load ASM kernel module

/usr/sbin/oracleasm init

Do the same on node2.

—————————Disk Creation

# Creating ASM disks

[root@node1 ~]# /usr/sbin/oracleasm createdisk CRSVOL /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@node1 ~]# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@node1 ~]# /usr/sbin/oracleasm createdisk DISK2 /dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@node1 ~]# /usr/sbin/oracleasm createdisk DISK3 /dev/sdd1
Writing disk header: done
Instantiating disk: done

You don’t need to do the above steps on node2.(Node2 may require reboot)

# List disks

[root@node1 ~]# /usr/sbin/oracleasm listdisks
CRSVOL
DISK1
DISK2
DISK3

# Testing disk discovery, that is used by DBCA

[root@node1 ~]# oracleasm-discover
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library – Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:CRSVOL [6281352 blocks (3216052224 bytes), maxio 512]
Discovered disk: ORCL:DISK1 [20964762 blocks (10733958144 bytes), maxio 512]
Discovered disk: ORCL:DISK2 [20964762 blocks (10733958144 bytes), maxio 512]
Discovered disk: ORCL:DISK3 [20964762 blocks (10733958144 bytes), maxio 512]

2.3.13 Check ping between nodes.

From node1: ping node2 and node2-priv

[root@node1 ~]# ping -c 1 node2
PING node2 (192.168.34.151) 56(84) bytes of data.
64 bytes from node2 (192.168.34.151): icmp_seq=1 ttl=64 time=1.51 ms

— node2 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.511/1.511/1.511/0.000 ms
[root@node1 ~]# ping -c 1 node2-priv
PING node2-priv (10.10.2.51) 56(84) bytes of data.
64 bytes from node2-priv (10.10.2.51): icmp_seq=1 ttl=64 time=2.76 ms

— node2-priv ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.763/2.763/2.763/0.000 ms

From node2: ping node1 and node1-priv

[root@node2 ~]# ping -c1  node1
PING node1 (192.168.34.150) 56(84) bytes of data.
64 bytes from node1 (192.168.34.150): icmp_seq=1 ttl=64 time=0.580 ms

— node1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.580/0.580/0.580/0.000 ms
[root@node2 ~]# ping -c1  node1-priv
PING node1-priv (10.10.2.50) 56(84) bytes of data.
64 bytes from node1-priv (10.10.2.50): icmp_seq=1 ttl=64 time=0.560 ms

— node1-priv ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.560/0.560/0.560/0.000 ms

2.3.13  Create SSH connectivity manually

From both nodes, install ssh and telnet clients:

[root@node1 Server]# cd /0/Server/
[root@node1 Server]# rpm -Uvh openssh-clients-4.3p2-41.el5.x86_64.rpm
warning: openssh-clients-4.3p2-41.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:openssh-clients        ########################################### [100%]

[root@node1 Server]# rpm -Uvh telnet-0.17-39.el5.x86_64.rpm
warning: telnet-0.17-39.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:telnet                 ########################################### [100%]

[root@node1 Server]# rpm -Uvh openssh-server-4.3p2-41.el5.x86_64.rpm
warning: openssh-server-4.3p2-41.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
package openssh-server-4.3p2-41.el5.x86_64 is already installed

[root@node1 Server]# rpm -qa | grep ssh
openssh-server-4.3p2-41.el5
openssh-askpass-4.3p2-41.el5
openssh-clients-4.3p2-41.el5
openssh-4.3p2-41.el5

From node1:

[root@node1 ~]# su – oracle
[oracle@node1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Created directory ‘/home/oracle/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
91:c3:7b:dc:20:c0:74:83:8a:f2:73:de:04:42:c8:fe oracle@node1
[oracle@node1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
f5:79:ab:dd:05:a7:42:48:25:00:27:2a:9d:a5:e2:e2 oracle@node1

From node2:

[root@node2 ~]# su – oracle
[oracle@node2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Created directory ‘/home/oracle/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
0d:8c:c2:24:1c:32:0e:bf:db:49:e1:e3:3c:71:ef:64 oracle@node2
[oracle@node2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
1d:52:a7:20:b4:54:4b:c9:12:e1:65:ca:08:a8:e9:de oracle@node2

From node1:

[oracle@node1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[oracle@node1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[oracle@node1 ~]$ ssh node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host ‘node2 (192.168.34.151)’ can’t be established.
RSA key fingerprint is 19:1e:e6:3a:cc:14:69:87:22:35:11:6d:aa:e6:dc:15.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node2,192.168.34.151’ (RSA) to the list of known hosts.
oracle@node2’s password:
[oracle@node1 ~]$ ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
oracle@node2’s password:
[oracle@node1 ~]$ scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys
oracle@node2’s password:
authorized_keys                               100% 2594     2.5KB/s   00:00

———————————————Check that you are not prompted for the password

From node1 and node2:

[oracle@node1 ~]$ ssh node1 date
The authenticity of host ‘node1 (192.168.34.150)’ can’t be established.
RSA key fingerprint is 24:a1:fb:71:ce:8a:9c:2d:6c:42:89:6f:e5:74:b2:f2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node1,192.168.34.150’ (RSA) to the list of known hosts.
Tue Sep  4 12:09:53 GET 2012
[oracle@node1 ~]$ ssh node2 date
Tue Sep  4 12:10:01 GET 2012
[oracle@node1 ~]$ ssh node1-priv date
The authenticity of host ‘node1-priv (10.10.2.50)’ can’t be established.
RSA key fingerprint is 24:a1:fb:71:ce:8a:9c:2d:6c:42:89:6f:e5:74:b2:f2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node1-priv,10.10.2.50’ (RSA) to the list of known hosts.
Tue Sep  4 12:10:11 GET 2012
[oracle@node1 ~]$ ssh node2-priv date
The authenticity of host ‘node2-priv (10.10.2.51)’ can’t be established.
RSA key fingerprint is 19:1e:e6:3a:cc:14:69:87:22:35:11:6d:aa:e6:dc:15.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node2-priv,10.10.2.51’ (RSA) to the list of known hosts.
Tue Sep  4 12:10:23 GET 2012

Check one more time to ensure that you are not prompted for the password. From node1 and node2.

[oracle@node1 ~]$ ssh node1 date
Tue Sep  4 12:14:57 GET 2012
[oracle@node1 ~]$ ssh node2 date
Tue Sep  4 12:15:01 GET 2012
[oracle@node1 ~]$ ssh node1-priv date
Tue Sep  4 12:15:06 GET 2012
[oracle@node1 ~]$ ssh node2-priv date
Tue Sep  4 12:15:12 GET 2012

Do the same steps with grid user:

From node1:

[root@node1 ~]# su – grid
[grid@node1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory ‘/home/grid/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
e1:c2:55:3e:fc:01:75:64:f7:69:72:f6:9d:1e:08:0c grid@node1
[grid@node1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
26:36:b8:e4:ac:d5:62:91:f3:46:8a:ac:65:73:ec:6a grid@node1

From node2:

[root@node2 ~]# su – grid
[grid@node2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory ‘/home/grid/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
f7:54:a6:21:e4:36:a8:9a:e7:93:fb:35:87:70:cc:05 grid@node2
[grid@node2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
33:36:04:e3:95:0e:b6:d8:92:67:a1:4e:e4:da:8f:26 grid@node2

From node1:

[grid@node1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[grid@node1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[grid@node1 ~]$ ssh node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host ‘node2 (192.168.34.151)’ can’t be established.
RSA key fingerprint is 19:1e:e6:3a:cc:14:69:87:22:35:11:6d:aa:e6:dc:15.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node2,192.168.34.151’ (RSA) to the list of known hosts.
grid@node2’s password:
[grid@node1 ~]$ ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
grid@node2’s password:
[grid@node1 ~]$ scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys
grid@node2’s password:
authorized_keys

———————————————Check that you are not prompted for the password

From node1 and node2:

[grid@node1 ~]$ ssh node1 date
The authenticity of host ‘node1 (192.168.34.150)’ can’t be established.
RSA key fingerprint is 24:a1:fb:71:ce:8a:9c:2d:6c:42:89:6f:e5:74:b2:f2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node1,192.168.34.150’ (RSA) to the list of known hosts.
Tue Sep  4 12:44:31 GET 2012
[grid@node1 ~]$ ssh node2 date
Tue Sep  4 12:44:37 GET 2012
[grid@node1 ~]$ ssh node1-priv date
The authenticity of host ‘node1-priv (10.10.2.50)’ can’t be established.
RSA key fingerprint is 24:a1:fb:71:ce:8a:9c:2d:6c:42:89:6f:e5:74:b2:f2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node1-priv,10.10.2.50’ (RSA) to the list of known hosts.
Tue Sep  4 12:44:47 GET 2012
[grid@node1 ~]$ ssh node2-priv date
The authenticity of host ‘node2-priv (10.10.2.51)’ can’t be established.
RSA key fingerprint is 19:1e:e6:3a:cc:14:69:87:22:35:11:6d:aa:e6:dc:15.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node2-priv,10.10.2.51’ (RSA) to the list of known hosts.
Tue Sep  4 12:44:59 GET 2012

Check one more time to ensure that you are not prompted for the password. From node1 and node2.

[grid@node1 ~]$ ssh node1 date
Tue Sep  4 12:46:57 GET 2012
[grid@node1 ~]$ ssh node2 date
Tue Sep  4 12:47:02 GET 2012
[grid@node1 ~]$ ssh node1-priv date
Tue Sep  4 12:47:08 GET 2012
[grid@node1 ~]$ ssh node2-priv date
Tue Sep  4 12:47:12 GET 2012

2.4 Oracle Grid Installation

Download Grid software installation. And locate it on the first node.

Download link: http://download.oracle.com/otn/linux/oracle11g/R2/linux.x64_11gR2_grid.zip

Login as a grid user.

# Unzip and install

mkdir  /home/grid/myinstall
unzip /tmp/linux.x64_11gR2_grid.zip  -d  /home/grid/myinstall

cd /home/grid/myinstall
chmod -R 777 *
cd 20120613/
chown -R grid:oinstall *

First of all, run runcluvfy.sh script  to ensure that pre-requisites are satisfied.

grid@node1 20120613]$ ./runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose

No errors were returned from the script above:

Pre-check for cluster services setup was successful

But if you have some failures like some packages are missing, correct them by installing before continue.

Now run runInstaller.sh . Don’t forget to connect as grid user from the first node.

[grid@node1 20120613]$ ./runInstaller

Virtual Machine_Install_and_Configure_Grid_Infrastructure

Virtual Machine_Typical_Installation_Grid

Add the second node:

Virtual Machine_Grid_Installation_Cluster_Configuration

Virtual Machine_Grid_Installation_Cluster_Configuration

Virtual Machine_Grid_Installation_location

Virtual Machine_Grid_Installation_ASM_Diskgroup

Virtual Machine_Grid_Installation_Inventory

Virtual Machine_Grid_Installation_Summary

Virtual Machine_Grid_Installation_orainstRoot.sh_root.sh

From node1:

[grid@node1 ~]$ su –
Password:
[root@node1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

From node2:

[root@node2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

From node1(wait until completes):

[root@node1 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin …
Copying oraenv to /usr/local/bin …
Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-09-04 14:59:27: Parsing the host name
2012-09-04 14:59:27: Checking for super user privileges
2012-09-04 14:59:27: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘node1’
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘node1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2676: Start of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘node1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘node1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘node1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘node1’
CRS-2676: Start of ‘ora.diskmon’ on ‘node1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘node1’
CRS-2676: Start of ‘ora.ctssd’ on ‘node1’ succeeded

ASM created and started successfully.

DiskGroup CRS created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-2672: Attempting to start ‘ora.crsd’ on ‘node1’
CRS-2676: Start of ‘ora.crsd’ on ‘node1’ succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 37e0beba09a14f4cbf7e8ba1524e1314.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
—  —–    —————–                ——— ———
1. ONLINE   37e0beba09a14f4cbf7e8ba1524e1314 (ORCL:CRSVOL) [CRS]
Located 1 voting disk(s).
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘node1’
CRS-2677: Stop of ‘ora.crsd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘node1’
CRS-2677: Stop of ‘ora.asm’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘node1’
CRS-2677: Stop of ‘ora.ctssd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssdmonitor’ on ‘node1’
CRS-2677: Stop of ‘ora.cssdmonitor’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘node1’
CRS-2677: Stop of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘node1’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘node1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘node1’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘node1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘node1’
CRS-2676: Start of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘node1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘node1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘node1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘node1’
CRS-2676: Start of ‘ora.diskmon’ on ‘node1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘node1’
CRS-2676: Start of ‘ora.ctssd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘node1’
CRS-2676: Start of ‘ora.asm’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘node1’
CRS-2676: Start of ‘ora.crsd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.evmd’ on ‘node1’
CRS-2676: Start of ‘ora.evmd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘node1’
CRS-2676: Start of ‘ora.asm’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.CRS.dg’ on ‘node1’
CRS-2676: Start of ‘ora.CRS.dg’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.registry.acfs’ on ‘node1’
CRS-2676: Start of ‘ora.registry.acfs’ on ‘node1’ succeeded

node1     2012/09/04 15:08:38     /u01/app/11.2.0/grid/cdata/node1/backup_20120904_150838.olr
Preparing packages for installation…
ls: /usr/sbin/smartctl: No such file or directory
/usr/sbin/smartctl not found.
error: %pre(cvuqdisk-1.0.7-1.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster … succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB.   Actual 4095 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
‘UpdateNodeList’ was successful.

If script execution was unsuccessful, then you are able to de-configure it by the following way(if script was successfully run then skip this step ):

[root@node1 ~] cd /u01/app/11.2.0/grid/crs/install
[root@node1 install]# perl rootcrs.pl -deconfig -force

From node2:

[root@node2 ~]# /u01/app/11.2.0/grid/root.sh

….

Virtual Machine_Grid_Installation_OC_Verification_Failed

Click OK.

SCAN verification failed because we are not using DNS.

Virtual Machine_Grid_Installation_Progress

Virtual Machine_Grid_Installation_Finish

Check cluster status for both nodes:

[grid@node1 ~]$ crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

[grid@node1 ~]$ olsnodes
node1
node2

[grid@node1 ~]$ crs_stat -t -v
Name           Type           R/RA   F/FT   Target    State     Host
———————————————————————-
ora.CRS.dg     ora….up.type 0/5    0/     ONLINE    ONLINE    node1
ora….ER.lsnr ora….er.type 0/5    0/     ONLINE    ONLINE    node1
ora….N1.lsnr ora….er.type 0/5    0/0    ONLINE    ONLINE    node1
ora.asm        ora.asm.type   0/5    0/     ONLINE    ONLINE    node1
ora.eons       ora.eons.type  0/3    0/     ONLINE    ONLINE    node1
ora.gsd        ora.gsd.type   0/5    0/     OFFLINE   OFFLINE
ora….network ora….rk.type 0/5    0/     ONLINE    ONLINE    node1
ora….SM1.asm application    0/5    0/0    ONLINE    ONLINE    node1
ora….E1.lsnr application    0/5    0/0    ONLINE    ONLINE    node1
ora.node1.gsd  application    0/5    0/0    OFFLINE   OFFLINE
ora.node1.ons  application    0/3    0/0    ONLINE    ONLINE    node1
ora.node1.vip  ora….t1.type 0/0    0/0    ONLINE    ONLINE    node1
ora….SM2.asm application    0/5    0/0    ONLINE    ONLINE    node2
ora….E2.lsnr application    0/5    0/0    ONLINE    ONLINE    node2
ora.node2.gsd  application    0/5    0/0    OFFLINE   OFFLINE
ora.node2.ons  application    0/3    0/0    ONLINE    ONLINE    node2
ora.node2.vip  ora….t1.type 0/0    0/0    ONLINE    ONLINE    node2
ora.oc4j       ora.oc4j.type  0/5    0/0    OFFLINE   OFFLINE
ora.ons        ora.ons.type   0/3    0/     ONLINE    ONLINE    node1
ora….ry.acfs ora….fs.type 0/5    0/     ONLINE    ONLINE    node1
ora.scan1.vip  ora….ip.type 0/0    0/0    ONLINE    ONLINE    node1

Run asmca to create DATA diskgroup:

[grid@node1 ~]$  asmca

Virtual Machine_Grid_Installation_asmca

Click Create.

Virtual Machine_Grid_Installation_asmca_Create_Disk

Virtual Machine_Grid_Installation_asmca

Click Exit.

2.4 Oracle Database Installation

Connect as an oracle user.

Download Link: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html?ssSourceSiteId=ocomen

Unzip downloaded file to /home/oracle/myinstall.

[oracle@node1 linux.x64_11gR2_database]$ ./runInstaller

Virtual Machine_Oracle_Database_Installation

Virtual Machine_Oracle_Database_Installation

Click Yes.

Virtual Machine_Oracle_Database_Installation_Create_and_Configure_Database

Virtual Machine_Oracle_Database_Installation_Server_Class

Virtual Machine_Oracle_Database_Installation_Rac

Virtual Machine_Oracle_Database_Installation_Typical

Virtual Machine_Oracle_Database_Installation_Configuration

Virtual Machine_Oracle_Database_Installation_Summary

After DBCA completes its job, there will appear the following window:

Virtual Machine_Oracle_Database_Installation_root.sh

From node1:

[oracle@node1 orcl]$ su –
Password:
[root@node1 ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file “dbhome” already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin …
The file “oraenv” already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin …
The file “coraenv” already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin …

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

From node2:

[root@node2 ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file “dbhome” already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin …
The file “oraenv” already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin …
The file “coraenv” already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y

Copying coraenv to /usr/local/bin …

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

Virtual Machine_Oracle_Database_Installation_Finish

To check:

From node1:

[oracle@node1 ~]$ echo $ORACLE_SID
orcl1
[oracle@node1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 4 18:26:10 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select instance_name from v$instance;

INSTANCE_NAME
—————-
orcl1

SQL> create table test(a number);

Table created.

SQL> insert into test values(1);

1 row created.

SQL> commit;

Commit complete.

From node2:

[oracle@node2 ~]$ echo $ORACLE_SID
orcl2
[oracle@node2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 4 18:29:49 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select instance_name from v$instance;

INSTANCE_NAME
—————-
orcl2

SQL> select * from test;

A
———-
1

Good Luck!

Install Oracle Database 11g on Linux with ASM

Let’s assume that you have already installed Linux on your server. And disks are already bounded to the server.

1. Creating OS groups and users.

#Creating groups for Grid Infrastructure

groupadd asmadmin
groupadd asmdba
groupadd asmoper

#Creating groups for Oracle Software

groupadd oinstall
groupadd dba
groupadd oper

#Creating user for Grid Infrastructure

useradd -g oinstall -G dba,asmadmin,asmdba,asmoper -d /home/grid grid

#Creating user for Oracle Software

useradd -g oinstall -G dba,oper,asmdba -d /home/oracle oracle

#Setting password for users

passwd grid
passwd oracle

2. Creating necessary directories

mkdir -p /u01/app/grid
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01

3. Creating .bash_profile-s

#For Oracle user

su – oracle
vi .bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=orcl; export ORACLE_SID

ORACLE_UNQNAME=orcl; export ORACLE_UNQNAME

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export ORACLE_HOME

ORACLE_TERM=xterm; export ORACLE_TERM

NLS_DATE_FORMAT=”DD-MON-YYYY HH24:MI:SS”
export NLS_DATE_FORMAT

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

THREADS_FLAG=native; export THREADS_FLAG

export TEMP=/tmp
export TMPDIR=/tmp

umask 022

#For Grid user

su – grid
vi .bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

ORACLE_SID=+ASM; export ORACLE_SID

JAVA_HOME=/usr/local/java; export JAVA_HOME

ORACLE_BASE=/u01/app/grid; export ORACLE_BASE

ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME

ORACLE_TERM=xterm; export ORACLE_TERM

NLS_DATE_FORMAT=”DD-MON-YYYY HH24:MI:SS”; export NLS_DATE_FORMAT

TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11

PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH

THREADS_FLAG=native; export THREADS_FLAG

export TEMP=/tmp
export TMPDIR=/tmp

umask 022

4. Setting resource limits

Edit the following files:

# /etc/security/limits.conf

[root@orcl ~]# cat >> /etc/security/limits.conf <<EOF
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536

oracle soft nproc 2047
oracle  hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF

# /etc/pam.d/login

[root@orcl ~]# cat >> /etc/pam.d/login <<EOF
session required pam_limits.so
EOF

# /etc/profile

[root@orcl ~]# cat >> /etc/profile <<EOF
if [ \$USER = “oracle” ] || [ \$USER = “grid” ]; then
if [ \$SHELL = “/bin/ksh” ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF

# /etc/csh.login

[root@orcl ~]# cat >> /etc/csh.login <<EOF
if ( \$USER == “oracle” || \$USER == “grid” )
then
limit maxproc 16384
limit descriptors 65536
endif
EOF

# /etc/hosts

[root@orcl ~]#  vi /etc/hosts

127.0.0.1 localhost.localdomain localhost
192.168.34.150 orcl

5. Installing ASMlib and creating disk groups

Download link: http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

# To know your kernel version

uname -r
2.6.18-194.el5xen

# Download the following files

oracleasm-support-2.1.7-1.el5.x86_64.rpm
oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm

oracleasmlib-2.0.4-1.el5.x86_64.rpm

# Installing

rpm -Uvh oracleasm-support-2.1.7-1.el5.x86_64.rpm
rpm -Uvh oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
rpm -Uvh oracleasmlib-2.0.4-1.el5.x86_64.rpm

# Configure ASM

/usr/sbin/oracleasm configure -i

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

# Load ASM kernel module

/usr/sbin/oracleasm init

##Partition available disks for ASM

#list available disks and partitions

fdisk -l

#partitioning

fdisk /dev/sdb

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-52216, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-52216, default 52216):
Using default value 52216

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

fdisk /dev/sdc

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-52216, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-52216, default 52216): +100M

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

# Restart the server

init 6

# Creating ASM disks

/usr/sbin/oracleasm createdisk VOL1 /dev/sdb1
/usr/sbin/oracleasm createdisk CRSVOL /dev/sdc1

# List disks

/usr/sbin/oracleasm listdisks
VOL1
CRSVOL

# Testing disk discovery, that is used by DBCA

oracleasm-discover

Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library – Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:CRSVOL [208782 blocks (106896384 bytes), maxio 512]
Discovered disk: ORCL:VOL1 [838849977 blocks (429491188224 bytes), maxio 512]

6. Installing Oracle Grid Infrastructure

Download link: http://download.oracle.com/otn/linux/oracle11g/R2/linux.x64_11gR2_grid.zip

Login as a grid user.

# Unzip and install

mkdir  /home/grid/myinstall
unzip /tmp/linux.x64_11gR2_grid.zip  -d  /home/grid/myinstall

cd /home/grid/myinstall
chmod -R 777 *
./runInstaller

Choose the following options:

6.1. Install and Configure Grid Infrastructure for a Standalone Server

6.2.  English

6.3. English

6.4.
Disk Group Name: CRS
Redundancy: External
Add Disks->Candidate Disks, choose just  ORCL:CRSVOL

6.5. Speciify password(s).

6.6.
ASM Database Administrator(OSDBA) Group : asmdba
ASM Instance Administration Operator(OSOPER) Group: asmoper
ASM Instance Administrator(OSASM) Group: asmadmin

6.7.
Oracle Base: /u01/app/grid
Software Location: /u01/app/11.2.0/grid

6.8. On the pop-up windows press Yes.

6.9. Inventory Directory: /u01/app/oraInventory

6.10. On the prerequisite checks page, there will be failed checks. To solve:

6.10.1 Press Fix & Check Again
6.10.2 Login as root user and run the following:

[root@orcl ~]# /tmp/CVU_11.2.0.1.0_grid/runfixup.sh

6.10.3 Press OK on the Execute Fixup window

Left just libaio* and unixODBC* packages that should be installed. To solve:

6.10.4 Mount Oracle Enterprise Linux installation disk and go to the Server folder, where rpms are located.

[root@orcl Server]# rpm -Uvh libaio-devel-0.3.106-5.i386.rpm
warning: libaio-devel-0.3.106-5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:libaio-devel           ########################################### [100%]
[root@orcl Server]# rpm -Uvh libaio-devel-0.3.106-5.x86_64.rpm
warning: libaio-devel-0.3.106-5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:libaio-devel           ########################################### [100%]
[root@orcl Server]# rpm -Uvh unixODBC-2.2.11-7.1.i386.rpm
warning: unixODBC-2.2.11-7.1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:unixODBC               ########################################### [100%]
[root@orcl Server]# rpm -Uvh unixODBC-2.2.11-7.1.x86_64.rpm
warning: unixODBC-2.2.11-7.1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:unixODBC               ########################################### [100%]
[root@orcl Server]# rpm -Uvh unixODBC-devel-2.2.11-7.1.i386.rpm
warning: unixODBC-devel-2.2.11-7.1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:unixODBC-devel         ########################################### [100%]
[root@orcl Server]# rpm -Uvh unixODBC-devel-2.2.11-7.1.x86_64.rpm
warning: unixODBC-devel-2.2.11-7.1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing…                ########################################### [100%]
1:unixODBC-devel         ########################################### [100%]

6.10. 5   Press Check Again.(There should not be any error)

6.11.

[root@orcl ~]# /u01/app/oraInventory/orainstRoot.sh
[root@orcl ~]# /u01/app/11.2.0/grid/root.sh

Press enter…

7. Creating ASM disk groups for database

Connect as a grid user.

[grid@orcl ~]$ asmca

7.1 Press Create button.

7.2
Disk Group Name: DATA
Redundancy: External(None)
Select Member Disks->Show Eligible, choose ORCL:VOL1

8. Installing Database, create instance on ASM

Connect as an oracle user.

Download Link: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html?ssSourceSiteId=ocomen

Unzip downloaded file to /home/oracle/myinstall.

[oracle@orcl linux.x64_11gR2_database]$ ./runInstaller

8.1 Uncheck “I with to receive security updates via My Oracle Support”

8.2 On the pop-up window choose yes.

8.3 Install database software only

8.4 Single instance database installation

8.5 English

8.6 Enterprise Edition (4.29GB)

8.7
Oracle Base: /u01/app/oracle
Software Location:  /u01/app/oracle/product/11.2.0/db_1

8.8
Database Administrator(OSDBA) Group: dba
Database Operator(OSOPER) Group: oper

8.9 I have some failed checks that are fixable so I press Fix & Check Again button, login as a root user and run:

[root@orcl ~]# /tmp/CVU_11.2.0.1.0_oracle/runfixup.sh

Click OK on the Execute Fixup window.

8.10

[root@orcl ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Press Enter…
y
y
y

9. Run DBCA as an oracle user.

[oracle@orcl ~]$ dbca

and follow the steps (choose ASM instead of File System option)

That’s all.