Oracle, Linux, AWS, Azure, GCP
June 15, 2022 Leave a comment
After installing/starting the tiger VNC server, and connecting using VNC Viewer terminal window is not displayed.
# yum install tigervnc-server
Install xterm:
# yum install xterm
Kill old VNC process and start again:
$ vncserver -kill :1
$ vncserver -geometry 1024x1024
Reconnect using VNC viewer, terminal should be displayed automatically.
April 21, 2022 Leave a comment
There are several ways to do that, I found the simplest (I hope so) and want to share it with you:
0. Create repo file:
# vi /etc/yum.repos.d/google-chrome.repo [google-chrome] name=google-chrome baseurl=https://dl.google.com/linux/chrome/rpm/stable/x86_64 enabled=1 gpgcheck=1 gpgkey=https://dl.google.com/linux/linux_signing_key.pub
1. Enable repo ol7_optional_latest for vulkan dependency:
# yum-config-manager --enable ol7_optional_latest
2. Install google-chrome-stable package:
# yum install google-chrome-stable -y
3. Run:
$ google-chrome
Or in the background:
$ google-chrome &
The window will come up in VNC or X Window whichever you’ve configured before.
March 22, 2022 Leave a comment
There are several ways to fulfill the same task, I am providing one of them.
# uname -r 5.4.17-2036.101.2.el7uek.x86_64
2. Find all available kernels in your system and locate their index number:
# grubby --info=ALL index=0 kernel=/boot/vmlinuz-5.4.17-2036.101.2.el7uek.x86_64 args="ro console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300 numa=off transparent_hugepage=never net.ifnames=0" root=/dev/mapper/rootvg-rootlv initrd=/boot/initramfs-5.4.17-2036.101.2.el7uek.x86_64.img title=Oracle Linux Server 7.9, with Unbreakable Enterprise Kernel 5.4.17-2036.101.2.el7uek.x86_64 index=1 kernel=/boot/vmlinuz-3.10.0-1160.42.2.el7.x86_64 args="ro console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300 numa=off transparent_hugepage=never net.ifnames=0" root=/dev/mapper/rootvg-rootlv initrd=/boot/initramfs-3.10.0-1160.42.2.el7.x86_64.img title=Oracle Linux Server 7.9, with Linux 3.10.0-1160.42.2.el7.x86_64 index=2 kernel=/boot/vmlinuz-0-rescue-d3dd3af16fd242cebb997c6041d68ad3 args="ro console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300 numa=off transparent_hugepage=never net.ifnames=0" root=/dev/mapper/rootvg-rootlv initrd=/boot/initramfs-0-rescue-d3dd3af16fd242cebb997c6041d68ad3.img
3. Check currently loaded kernel index using grubby tool (actually, we could find the same from 1st and 2nd steps, but let’s do one more time):
# grubby --default-index 0
4. Change the default kernel, in my case I want to set it to vmlinuz-3.10.0-1160.42.2.el7.x86_64 and it’s index number is 1:
# grubby --set-default 1
5. Reboot the system and check the kernel again:
# reboot # uname -r 3.10.0-1160.42.2.el7.x86_64
December 17, 2021 Leave a comment
Yes, it can be changed without impacting the business operations.
Oracle source: Doc ID 2269728.1
RHEL source: https://access.redhat.com/solutions/437043
========================Additional information========================
Additional explanation, why you may need to change that value.
Alert log shows:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O Linux-x86_64 Error: 2: No such file or directory Additional information: 3 Additional information: 128 Additional information: 139817340277512
The problem is caused by a lower than recommended value for aio-max-nr. The current value is 1048576, but recommended is 3145728.
Set fs.aio-max-nr to 3145728 in /etc/sysctl.conf.
# grep fs.aio-max-nr /etc/sysctl.conf fs.aio-max-nr = 3145728
Make a new value effective:
# sysctl -p /etc/sysctl.conf
Verify the new value:
# sysctl -a|grep fs.aio-max-nr fs.aio-max-nr = 3145728
December 8, 2021 Leave a comment
Connecting via ssh to the newly created host causes error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:AxfpHOVc8NP2OYPGce92HMa5LADDQj2V98ZKgoQHFGU.
Please contact your system administrator.
Add correct host key in /Users/mari/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/mari/.ssh/known_hosts:315
ECDSA host key for 52.1.130.91 has changed and you have requested strict checking.
Host key verification failed.I had another server with the same Public IP, so when I connected to the old saver the host identification has been saved in known_hosts. After a while I have removed old server and created a new one and assigned the PIP. The host identification has changed, but old entries were still saved in known_hosts.
Open /Users/mari/.ssh/known_hosts and delete only the line containing mentioned IP (52.1.130.91 in my case), save file and retry the connection.
It should work now.
November 26, 2021 Leave a comment
Databases were crashed and alert logs were showing errors:
Fri Nov 12 13:23:39 2021
Process startup failed, error stack:
Errors in file /app/oracle/diag/rdbms/orcl/orcl/trace/orcl_psp0_25852.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
We’ve implemented a procedure to take process list (by ps -ef), so output during that time was the following:
oracle 592 1 0 13:57 ? 00:00:00 [oracle] <defunct> oracle 593 1 0 13:55 ? 00:00:00 [oracle] <defunct> oracle 615 1 0 13:57 ? 00:00:00 [oracle] <defunct> oracle 618 1 0 13:57 ? 00:00:00 [oracle] <defunct> ...
Not only Oracle, but sshd and some other processes were also experiencing the same:
oracle 22335 22331 0 13:52 ? 00:00:00 [ps] <defunct> oracle 22336 22331 0 13:52 ? 00:00:00 [grep] <defunct> oracle 22338 22331 0 13:52 ? 00:00:00 [grep] <defunct> oracle 14389 1 0 13:24 ? 00:00:00 [sshd] <defunct> oracle 15852 1 0 13:23 ? 00:00:00 [sshd] <defunct>
A large amount of Zombie processes, causing applications to fail.
You may find a lot of recommendations about increasing kernel.pid_max, similarly ORA-27300: OS System Dependent Operation:fork Failed With Status: 11 (Doc ID 1546393.1). Of course, you can make this parameter unlimited, but this will not solve the problem, it will just postpone it.
The reason for high number of defunct processes is described here, https://access.redhat.com/solutions/2438581.
The parent process for our defunct processes was systemd (pid=1) and the version of it was systemd-219-19.el7.x86_64.
The solution is to update systemd to the latest version.
November 25, 2021 Leave a comment
If the number of files you are working on is big, then you need automation as soon as possible.
This post describes find -o option, which helps you work on differently named files when their number is big.
For example, if you want to output the content of files physical_block_size and logical_block_size located under /sys/block/*/queue, run the following:
# find /sys/block/*/queue -name physical_block_size -o -name logical_block_size | while read f ; do echo "$f $(cat $f)" ; done .. /sys/block/dm-0/queue/physical_block_size 4096 /sys/block/dm-0/queue/logical_block_size 512 /sys/block/dm-1/queue/physical_block_size 512 ...
Where -o means OR.
Useful when working on ASM disks.
September 8, 2021 1 Comment
I’ve recently heard about this tool , as it is said we are learning things until the death (:
Our company is saving sar output in a text file periodicly and after performance or other issues we need to analyze it’s output to find out which resource was busy and when.. analyzing text file is time-consuming and can also cause eye tension.
Output in sar:
00:00:01 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 00:10:01 all 3.14 0.00 2.43 1.64 0.00 0.00 0.60 0.00 0.00 92.20 00:10:01 0 3.64 0.00 2.33 4.10 0.00 0.00 1.10 0.00 0.00 88.83 00:00:01 scall/s badcall/s packet/s udp/s tcp/s hit/s miss/s sread/s swrite/s saccess/s sgetatt/s 00:10:01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ... 00:00:01 totsck tcpsck udpsck rawsck ip-frag tcp-tw 00:10:01 5682 656 1783 0 6 502 00:20:01 5651 668 1748 0 0 804
CPU, Network, Disk I/O, etc. activities are logged.
Same text file analyzed by ksar tool and graphycally displayed is the following:
Full list of items that can be seen graphycally are the following:
Now I will show all necessary information that is necessary to use this tool:
1. Download a pre-built jar from GitHub releases page.
2. Run jar on your computer:
java -jar ksar-5.2.4-b396_gf0680721-SNAPSHOT-all.jar
3. Click Data -> Load from a file…and choose output of sar in a text file
Full information about this tool: https://github.com/vlsi/ksar
August 23, 2021 Leave a comment
Problem:
I want to send email notification to my Gmail account from Linux server using Postfix client. Mails are not received and /var/log/maillog is full of the following error messages:
Aug 18 17:24:29 rac1 postfix/smtp[17580]: connect to gmail-smtp-in.l.google.com[74.125.69.27]:25: Connection timed out Aug 18 17:24:29 rac1 postfix/smtp[17580]: connect to gmail-smtp-in.l.google.com[2607:f8b0:4001:c0d::1a]:25: Network is unreachable Aug 18 17:24:59 rac1 postfix/smtp[17580]: connect to alt1.gmail-smtp-in.l.google.com[173.194.77.27]:25: Connection timed out Aug 18 17:25:29 rac1 postfix/smtp[17580]: connect to alt2.gmail-smtp-in.l.google.com[173.194.219.27]:25: Connection timed out
Solution:
Configure Postfix and Gmail account accordingly.
1. Confirm that the myhostname parameter is configured with your server’s FQDN:
# grep ^myhostname /etc/postfix/main.cf myhostname = rac1.example.com
2. Generate an App Password for Postfix:
Click on App passwords -> Select app dropdown -> choose Other (custom name) -> Enter “Postfix” -> click GENERATE.
Postfix app password is generated in yellow box, copy and save it (generated_password_goes_here will be changed by this value).
3. Fill SMTP Host, username, and password in /etc/postfix/sasl_passwd
# cat /etc/postfix/sasl_passwd smtp.gmail.com your_username@gmail.com:generated_password_goes_here
4. Create the hash db file
# postmap /etc/postfix/sasl_passwd
5. Configure the Postfix Relay Server:
# grep ^relayhost /etc/postfix/main.cf relayhost = [smtp.gmail.com]:587
6. To enable authentication, add the following parameters in /etc/postfix/main.cf
smtp_sasl_auth_enable = yes smtpd_tls_auth_only = yes smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd smtp_sasl_security_options = noanonymous smtp_tls_security_level = encrypt
7. Reload Postfix service:
# systemctl reload postfix
8. For sending test email, I use Flashgrid tool:
[root@rac1 ~]# flashgrid-node test-alerts FlashGrid 21.2.24.58935 #bb6005e9d66650d1996184c38d2fb8a2a78420a8 License: Active, Marketplace Licensee: Flashgrid Inc. Support plan: 24x7 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Test alerts were sent
The alert is received now:
DBA Knowledge Base 