Can fs.aio-max-nr be changed on a live system?

December 17, 2021 Leave a comment

Short answer:

Yes, it can be changed without impacting the business operations.
Oracle source: Doc ID 2269728.1
RHEL source: https://access.redhat.com/solutions/437043

========================Additional information========================

Additional explanation, why you may need to change that value.

Problem:

Alert log shows:

ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Additional information: 128
Additional information: 139817340277512

Reason:

The problem is caused by a lower than recommended value for aio-max-nr. The current value is 1048576, but recommended is 3145728.

Solution:

Set fs.aio-max-nr to 3145728 in /etc/sysctl.conf. 

# grep fs.aio-max-nr /etc/sysctl.conf
fs.aio-max-nr = 3145728

Make a new value effective:

# sysctl -p  /etc/sysctl.conf

Verify the new value:

# sysctl -a|grep fs.aio-max-nr
fs.aio-max-nr = 3145728

Filed under Linux, Oracle Enterprise Linux, OS Tagged with , , , , ,

THE SPECIFIED LOGFILE BLOCKSIZE (512) IS LESS THAN THE MEDIA SECTOR SIZE ON DISK (4096)

December 9, 2021 Leave a comment

Problem:

Alert log contains: 

WARNING! THE SPECIFIED LOGFILE BLOCKSIZE (512) IS LESS THAN THE MEDIA SECTOR SIZE ON DISK (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO READ-MODIFY-WRITE OPERATION

Reason/Reproduce:

The redo log groups have the default block size 512. So as the error mentions, the size of the redo log group member is less than the sector size attribute for the diskgroup.

This is reproducable when disk physical block size is 4096. In AWS environment, for example, r5b instances has this size for attached disks (this may change in future, need to be checked while reading this post).

1. Make sure attached disks has physical size 4096

[root@rac1 ~]# find /sys/block/*/queue -name physical_block_size | while read f ; do echo "$f $(cat $f)" ; done

/sys/block/nvme0n1/queue/physical_block_size 4096
/sys/block/nvme1n1/queue/physical_block_size 4096
/sys/block/nvme2n1/queue/physical_block_size 4096
/sys/block/nvme3n1/queue/physical_block_size 4096
/sys/block/nvme4n1/queue/physical_block_size 4096
/sys/block/nvme5n1/queue/physical_block_size 4096

2. Create diskgroup with the sector size 4096

CREATE DISKGROUP "DG4K" NORMAL REDUNDANCY                                                
 FAILGROUP "RAC1" DISK '/dev/flashgrid/rac1.xvdbd' NAME "RAC1$XVDBD" SIZE 10240M 
 FAILGROUP "RAC2" DISK '/dev/flashgrid/rac2.xvdbd' NAME "RAC2$XVDBD" SIZE 10240M 
 QUORUM FAILGROUP "RACQ" DISK '/dev/flashgrid/racq.xvdbz' NAME "RACQ$XVDBZ"      
 ATTRIBUTE 'au_size' = '4M', 'compatible.asm' = '19.0.0.0', 'compatible.rdbms' = '19.0.0.0', 'sector_size'='4096';

Make sure sector size is 4096 for this diskgroup:

SQL> select name,sector_size from v$asm_diskgroup;

NAME			       SECTOR_SIZE
------------------------------ -----------
DATA				       512
GRID				       512
DG4K				      4096
FRA				       512

Check log group block size:

SQL> select group#, blocksize from v$log;

    GROUP#  BLOCKSIZE
---------- ----------
	 1	  512
	 2	  512
	 3	  512
	 4	  512

Add a new logfile member on DG4K:

SQL> alter database add logfile member '+DG4K' to group 4;

Database altered.

Check alert log:

2021-12-09T15:32:04.533204+00:00
WARNING! THE SPECIFIED LOGFILE BLOCKSIZE (512) IS LESS THAN THE MEDIA SECTOR SIZE ON DISK (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO READ-MODIFY-WRITE OPERATIONS.
2021-12-09T15:32:21.080534+00:00
Completed: alter database add logfile member '+DG4K' to group 4

Solution:

You should create a new redo log group (not member, but group) with block size=4K:

SQL> alter database add logfile group 5 ('+DG4K') size 512M blocksize 4096;

Filed under DB Tagged with , ,

Make Oracle ASM voting file online

December 9, 2021 Leave a comment

Problem:

After changing the quorum node instance type, my cluster’s one of the voting file became offline:

[root@rac1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   49400dd2b39a4f12bf3c5fa677c056fe (/dev/flashgrid/rac2.xvdba) [GRID]
 2. ONLINE   4a6d94d206104fe6bfbe5435ac7f4586 (/dev/flashgrid/rac1.xvdba) [GRID]
 3. OFFLINE  faf99f5fd78f4f35bfe833bdd1d22b9a (/dev/flashgrid/racq.xvdba) [GRID]
Located 3 voting disk(s).

Solution:

Find out the ASM disk name which contains mentioned voting file, offline and online it:

SQL> select NAME from v$ASM_DISK where PATH='/dev/flashgrid/racq.xvdba';

NAME
------------------------------
RACQ$XVDBA

Offline the disk:

SQL> alter diskgroup GRID offline quorum disk "RACQ$XVDBA";

Diskgroup altered.

Online again:

SQL> alter diskgroup GRID online quorum disk "RACQ$XVDBA";

Diskgroup altered.

Check the status again:

SQL> !crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   49400dd2b39a4f12bf3c5fa677c056fe (/dev/flashgrid/rac2.xvdba) [GRID]
 2. ONLINE   4a6d94d206104fe6bfbe5435ac7f4586 (/dev/flashgrid/rac1.xvdba) [GRID]
 3. ONLINE   784f924d23c94f3fbf4287c5c6ef572c (/dev/flashgrid/racq.xvdba) [GRID]

Filed under ASM Tagged with , ,

REMOTE HOST IDENTIFICATION HAS CHANGED!

December 8, 2021 Leave a comment

Problem:

Connecting via ssh to the newly created host causes error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:AxfpHOVc8NP2OYPGce92HMa5LADDQj2V98ZKgoQHFGU.
Please contact your system administrator.
Add correct host key in /Users/mari/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/mari/.ssh/known_hosts:315
ECDSA host key for 52.1.130.91 has changed and you have requested strict checking.
Host key verification failed.

Reason:

I had another server with the same Public IP, so when I connected to the old saver the host identification has been saved in known_hosts. After a while I have removed old server and created a new one and assigned the PIP. The host identification has changed, but old entries were still saved in known_hosts.

Solution:

Open /Users/mari/.ssh/known_hosts and delete only the line containing mentioned IP (52.1.130.91 in my case), save file and retry the connection.
It should work now.

Filed under DB, Linux Tagged with , , ,