Part 2: ora.storage fails to start, ORA-01017

Problem:

One of our customers changed ASM password file by mistake and regarding other actions, we are not sure. After node restart, they encountered ora.storage startup issue on the second node.

CRS-2672: Attempting to start 'ora.storage' on 'orcl02'
ORA-01017: invalid username/password; logon denied
CRS-5055: unable to connect to an ASM instance because no ASM instance is running in the cluster
CRS-2883: Resource 'ora.storage' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
CRS-4000: Command Start failed, or completed with errors.

I have followed my blog post to recover ASM passwordfile and add CRSUSER__ASM_001. The CRS started successfully on the first node but it still was not able to start on the second.

Reason:

When we checked password for CRSUSER__ASM_001 on both nodes, we got different results:

[grid@orcl01 ~]$ crsctl get credmaint -path ASM/Self/0b5330fe4bdf6f6ebffb09beab078d6e -credtype userpass -id 0 -attr passwd -local 
zSZDts1PQx8v7gRrdmH1EjIpSBsAt
[grid@orcl02 ~]$ crsctl get credmaint -path ASM/Self/0b5330fe4bdf6f6ebffb09beab078d6e -credtype userpass -id 0 -attr passwd -local 
rHgulYGfY17Uxbb9Tbd9VF3yr2Kvr

Which is not normal and they must be the same. This was the reason CRS was not able to start on the second node, because ASM passwordfile for CRSUSER__ASM_001 had value zSZDts1PQx8v7gRrdmH1EjIpSBsAt

Solution:

Verify and fix the credentials:

If you are not able to set up root ssh passwordless connectivity, you can run the following command as grid. Note in that case you will get “credfix: could not delete crs credentials for jxrucJl3”, this is because the command was not run as root and old credentials were not deleted. But new credentials are successfully created.

[grid@orcl01 ~]$ asmcmd --nocp credverify
credverify: More than one credential in password file, please run 'credfix' to fix the credentials.
‚Äč
[grid@orcl01 ~]$ asmcmd --nocp credfix
credfix: Credentials for JXRUCJL3 not in password file, trying next credential.
op=addcrscreds wrap=/tmp/creds0.xml
credfix: Creating new credentials, no valid credentials in OCR.
credfix: New user CRSUSER__ASM_004 created.
op=credimport wrap=/tmp/creds0.xml olr=true force=true
credfix: OLR for orcl01 has been fixed if credentials were created incorrectly.
credfix: Starting SSH session on node orcl02.
credfix: OLR for orcl02 has been fixed if credentials were created incorrectly. Exiting SSH session.
op=delcrscreds crs_user=jxrucJl3
ASMCMD-8202: internal error:
credfix: could not delete crs credentials for jxrucJl3

It is recommended to setup passwordless ssh connectivity for root user and then run credfix as root to have clean configuration without old entries:

[root@rac1 ~]# asmcmd --nocp credfix
..

Installing linux.x64_193000_client gets java.library.path system variable is missing or invalid

Problem:

While installing linux.x64_193000_client.zip on Linux, getting the following error:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2022-06-15_07-48-19AM. Please wait ...[oracle@mariclient client]$ Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/u01/tmp
The java.library.path system variable is missing or invalid. Please set java.library.path with a correct value and retry the operation.
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader
	at oracle.sysman.oii.oiip.oiipg.OiipgBootstrap.isCleanMachine(OiipgBootstrap.java:484)
	at oracle.sysman.oii.oiii.OiiiInstallAreaControl.isCleanMachine(OiiiInstallAreaControl.java:3796)

Solution:

Install the following rpm and retry:

# yum install libnsl.x86_64

Terminal window xterm is not displayed in VNC

Problem:

After installing/starting the tiger VNC server, and connecting using VNC Viewer terminal window is not displayed.

# yum install tigervnc-server

Solution:

Install xterm:

# yum install xterm

Kill old VNC process and start again:

$ vncserver -kill :1
$ vncserver -geometry 1024x1024

Reconnect using VNC viewer, terminal should be displayed automatically.

RMAN backup on NFS v3 takes too much time

Problem:

I was trying to test RMAN backup on Azure blob storage with NFS v3 access. I have noticed a huge delay even when backing up a small controlfile.

The same backup on disk took 7sec, but on NFS 15min. While watching IO on NFS mountpoint using nfsiostat I saw that initial 13min there was no IO at all and actual backup was taken only at the last minute.

Another interesting thing while watching netstat output was the following:

[root@rac1 dbbackup2]# netstat -na|grep 10.0.0.
tcp 0 0 10.0.0.5:875 10.0.0.16:2048 ESTABLISHED
tcp 0 1 10.0.0.5:54997 10.0.0.16:2049 SYN_SENT

You can also turn debug on before running backup in RMAN and check the output:

RMAN> RUN
{
  ALLOCATE CHANNEL disk1 DEVICE TYPE DISK FORMAT '/dataload/%U';
  debug on;
  BACKUP current controlfile;
  debug off;
}

Solution:

Disable dNFS.

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk dnfs_off

Retry RMAN backup, in my case it took only 1sec:

...
piece handle=/dbbackup/0q0rruhd_26_1_1 tag=TAG20220425T202309 comment=NONE
channel disk1: backup set complete, elapsed time: 00:00:01
Finished backup at 25-APR-22

Monitor NFS mount point IO performance

Problem:

I am configuring RMAN backup of my Oracle database and redirecting backup sets to Azure Blob Storage with NFSv3 access (quite a new feature for this time and needs to be tested). But I don’t know what will be the writing performance for this type of storage.

Solution:

One of the useful tools is nfsiostat, we will test it in this blog post.

1. After mounting Azure Blob Storage to my database node as /dbbackup mount point, instead of triggering RMAN backup I’d prefer dd command at this time:

$  dd if=/dev/zero of=/dbbackup/myfile oflag=direct bs=1M count=512000

2. Run nfsiostat with interval 1sec and monitor values:

[oracle@rac1 data]$ nfsiostat 1

marirmanstorage.blob.core.windows.net:/marirmanstorage/dbbackup mounted on /dbbackup:

   op/s	   rpc bklog
   0.11	   0.00
read:      ops/s    kB/s      kB/op.    retrans	 avg RTT (ms)	avg exe (ms)
	   0.000    0.000     0.000     0 (0.0%) 0.000	        0.000
write:     ops/s    kB/s      kB/op     retrans	 avg RTT (ms)	avg exe (ms)
	   18.000   18437.977 1024.332  0 (0.0%) 55.500	        55.611

For information, interval specifies the amount of time in seconds between each report. The first report contains statistics for the time since each file system was mounted. Each subsequent report contains statistics collected during the interval since the previous report

3. Cancel dd operation, you will also get information about the speed :

[oracle@rac1 dbbackup]$  dd if=/dev/zero of=/dbbackup/myfile oflag=direct bs=1M count=512000
^C
124+0 records in
124+0 records out
130023424 bytes (130 MB) copied, 6.85939 s, 19.0 MB/s

More information about nfsiostat can of course be found using man nfsiostat.

Database Express Setup: This site can not be reached

Note: These steps are for TEST non-production databases only.

Configuring/enabling EM express on HTTPS should be simple, by running only:

SQL> exec DBMS_XDB_CONFIG.SETHTTPSPORT(5500);

But after configuring it you may still not be able to access the web page and show the error “This site can’t be reached”

The reason for this can be permissions on the wallet files:

[grid@rac1 ~]$ ll /u01/app/oracle/product/19.3.0/dbhome_1/admin/orcl/xdb_wallet
total 8
-rw------- 1 oracle asmadmin 3864 Apr 15 17:40 cwallet.sso
-rw------- 1 oracle asmadmin 3819 Apr 15 17:40 ewallet.p12

The correct permission is 600, but when database is on top of ASM with role separation, then group should also have read permission on these files:

[root@rac1 ~]# chmod 640 /u01/app/oracle/product/19.3.0/dbhome_1/admin/orcl/xdb_wallet/*

After changing it, the web page is displayed.

Install Google Chrome on Linux 7.9 using terminal

There are several ways to do that, I found the simplest (I hope so) and want to share it with you:

0. Create repo file:

# vi /etc/yum.repos.d/google-chrome.repo

[google-chrome]
name=google-chrome
baseurl=https://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl.google.com/linux/linux_signing_key.pub

1. Enable repo ol7_optional_latest for vulkan dependency:

# yum-config-manager --enable ol7_optional_latest

2. Install google-chrome-stable package:

# yum install google-chrome-stable -y

3. Run:

$ google-chrome

Or in the background:

$ google-chrome &

The window will come up in VNC or X Window whichever you’ve configured before.

Monitoring ASM disk performance using IOSTAT

iostat in asmcmd displays I/O statistics for Oracle ASM disks in mounted disk groups.

Connect to the database node via GI owner:

# su - grid

Run iostat with the following options (Reads & Writes are in bytes):

# asmcmd
ASMCMD> iostat -t -G FRA 5
Group_Name  Dsk_Name   Reads      Writes    Read_Time  Write_Time
FRA         RAC1$LUN3  585083392  98942464  94.659862  4.03044
FRA         RAC2$LUN3  1847296    98942464  .054822    4.134049
FRA         RACQ$LUN4  57344      24576     .035944    .018594

Group_Name  Dsk_Name   Reads      Writes  Read_Time  Write_Time
FRA         RAC1$LUN3  368640.00  0.00    0.01       0.00
FRA         RAC2$LUN3  0.00       0.00    0.00       0.00
FRA         RACQ$LUN4  0.00       0.00    0.00       0.00

Where
-t displays time statistics (Read_Time, Write_Time)
-G FRA displays statistics for the FRA diskgroup, change the diskgroup name according to your needs.
5 is a refresh interval. When the interval is specified then the value displayed (bytes or I/Os) is the difference between the previous and current values, not the total value. But if a refresh interval is not specified, the number displayed represents the total number of bytes or I/Os.

For synopsis and description about all available iostat options, run help:

ASMCMD> help iostat
iostat
        Displays I/O statistics for Oracle ASM disks in mounted disk groups.

Synopsis
        iostat [-et][--io] [--suppressheader] [--region] [-G <diskgroup>] [<interval>]

Description
        iostat lists disk group statistics using the V$ASM_DISK_STAT view.
        The options for the iostat command are described below.
        -e		- Displays error statistics (Read_Err, Write_Err).
        -G diskgroup	- Displays statistics for the disk group name.
        --suppressheader	- Suppresses column headings.
        --io		- Displays information in number of I/Os, instead
                          of bytes.
        -t		- Displays time statistics (Read_Time, Write_Time).
        --region	- Displays information for cold and hot disk regions
                          (Cold_Reads, Cold_Writes, Hot_Reads, Hot_Writes).
        interval	- Refreshes the statistics display based on the
                          interval value (seconds).
        The attribute descriptions for iostat command output are described
	below. To view the complete set of statistics for a disk group,
	use the V$ASM_DISK_STAT view.
        Group_Name	        Name of the disk group.
        Dsk_Name	        Name of the disk.
        Reads	        	Total number of bytes read from the disk.
				If the --io option is entered, then the value
				is displayed as number of I/Os.
        Writes	        	Total number of bytes written to the disk.
				If the --io option is entered, then the value
				is displayed as number of I/Os.
        Cold_Reads	        Total number of bytes read from the cold disk
				region. If the --io option is entered, then
				the value is displayed as number of I/Os.
        Writes	        	Total number of bytes written to the disk.
        Cold_Writes	        Total number of bytes written to the cold
				disk region. If the --io option is entered,
				then the value is displayed as number of I/Os.
        Hot_Reads	        Total number of bytes read from the hot
				disk region. If the --io option is entered,
				then the value is displayed as number of I/Os.
        Writes	        	Total number of bytes written to the disk.
        Cold_Writes	        Total number of bytes written to the cold
        Hot_Writes	        Total number of bytes written to the hot disk
				region. If the --io option is entered, then the
				value is displayed as number of I/Os.
        Read_Err	        Total number of failed I/O read requests for
				the disk.
        Write_Err	        Total number of failed I/O write requests for
				the disk.
        Read_Time	        Total I/O time (in seconds) for
				read requests for the disk if the
				TIMED_STATISTICS initialization parameter is
				set to TRUE (0 if set to FALSE).
        Write_Time	        Total I/O time (in seconds) for
				write requests for the disk if the
				TIMED_STATISTICS initialization parameter is
				set to TRUE (0 if set to FALSE).
        Writes	        	Total number of bytes written to the disk.
        Cold_Writes	        Total number of bytes written to the cold
        Hot_Writes	        Total number of bytes written to the hot disk
        If a refresh interval is not specified, the number displayed represents
        the total number of bytes or I/Os.  Ifa refresh interval is specified,
        then the value displayed (bytes or I/Os) is the difference between the
        previous and current values, not the total value.

Examples
        The following are examples of the iostat command. The first example
        displays disk I/O statistics for the data disk group in total number
        of bytes. The second example displays disk I/O statistics for the data
        disk group in total number of I/O operations.
        ASMCMD [+] > iostat -G data
        Group_Name  Dsk_Name   Reads       Writes
        DATA        DATA_0000  180488192   473707520
        DATA        DATA_0001  1089585152  469538816
        DATA        DATA_0002  191648256   489570304
        DATA        DATA_0003  175724032   424845824
        DATA        DATA_0004  183421952   781429248
        DATA        DATA_0005  1102540800  855269888
        DATA        DATA_0006  171290624   447662592
        DATA        DATA_0007  172281856   361337344
        DATA        DATA_0008  173225472   390840320
        DATA        DATA_0009  288497152   838680576
        DATA        DATA_0010  196657152   375764480
        DATA        DATA_0011  436420096   356003840
        ASMCMD [+] > iostat --io -G data
        Group_Name  Dsk_Name   Reads  Writes
        DATA        DATA_0000  2801   34918
        DATA        DATA_0001  58301  35700
        DATA        DATA_0002  3320   36345
        DATA        DATA_0003  2816   10629
        DATA        DATA_0004  2883   34850
        DATA        DATA_0005  59306  38097
        DATA        DATA_0006  2151   10129
        DATA        DATA_0007  2686   10376
        DATA        DATA_0008  2105   8955
        DATA        DATA_0009  9121   36713
        DATA        DATA_0010  3557   8596
        DATA        DATA_0011  17458  9269

DBT-06103 The port (1,521) is already used

Thanks Tornike Kupatadze for this testing case!

Problem:

During my OCA class, after successful 19c database software installation, we were creating a database using dbca and got the following error:

If the listener had already been configured we would have had an error DBT-06103 The port (5,500) is already used while configuring EM express. But still, the solution is the same.

We have checked and the port was not used:

# netstat -a |grep 1521

Reason:

The hostname is not reachable. The reason in our case was that /etc/hosts did not contain entries about this server.

Solution:

Qualify the hostname into the /etc/hosts:

After adding the above entry, we were able to continue.

ORA-15120: ASM file name ‘ORA-27090: Unable to reserve kernel resources f’ does not begin with the ASM prefix character

Problem:

The customer created 36 databases on the same server and while creating the 37th using dbca got the following error:

Reason:

fs.aio-max-nr value was set too low in /etc/sysctl.conf. In general, value 3145728 that was set in their case, suits many environments, but if the number of databases on the server increases then this parameter should be adjusted accordingly.

Solution:

The formula used while calculating the value for this parameter is the following:

aio-max-nr = no of process per DB * no of databases * 4096

In their case, the number of processes per DB was 1000, the number of databases that planned to be created was 80. Based on the above value should be:

aio-max-nr = 327680000
  1. Add/update value in /etc/sysctl.conf:
# vim /etc/sysctl.conf

fs.aio-max-nr = 327680000


2. Run /sbin/sysctl -p to immediately enforce the changes:

# sysctl -p 

Delete already created files and recreate the database, it will succeed this time.