One of our customer incorrectly changed fstab file and rebooted the OS. As a result, VM was not able to start. Fortunately, cloud where this VM was located supported serial console.
Solution:
We booted in single user mode through serial console and reverted the changes back. To boot in single user mode and update necessary file, do as follows:
Connect to the serial console and while OS is booting in a grub menu press e to edit the selected kernel:
Find line that starts with linux16 ( if you don’t see it press arrow down ), go to the end of this line and type rd.break.
Press ctrl+x.
Wait for a while and system will enter into single user mode:
During this time /sysroot is mounted in read only mode, you need to remount it in read write:
switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot
You can revert any changes back by updating any file, in our case we updated fstab:
sh-4.2# vim /etc/fstab
Restart VM:
sh-4.2# reboot -f
You are a real hero, because you rescued your system!
Oracle 18c GI configuration precheck was failing with the following error:
Summary of node specific errors
primrac2 - PRVG-11069 : IP address "169.254.0.2" of network interface "idrac" on the node "primrac2" would conflict with HAIP usage.
- Cause: One or more network interfaces have IP addresses in the range (169.254..), the range used by HAIP which can create routing conflicts.
- Action: Make sure there are no IP addresses in the range (169.254..) on any network interfaces.
primrac1 - PRVG-11069 : IP address "169.254.0.2" of network interface "idrac" on the node "primrac1" would conflict with HAIP usage.
- Cause: One or more network interfaces have IP addresses in the range (169.254..), the range used by HAIP which can create routing conflicts.
- Action: Make sure there are no IP addresses in the range (169.254..) on any network interfaces.
On each node additional network interface – named idrac was started with the ip address 169.254.0.2. I tried to set static ip address in /etc/sysconfig/network-scripts/ifcfg-idrac , also tried to bring the interface down – but after some time interface was starting up automatically and getting the same ip address.
Servers were configured by system administrator and was not clear why this module was there, we are not using iDRAC module and the only option that we had was to remove/uninstall that module. (configuring module should also be possible to avoid such situation, but we keep our servers as clean as possible without having unsed services)
Solution:
Uninstalled iDRAC module (also expained in the above pdf):
# rpm -e dcism
After uninstalling it idrac interface did not started anymore, so we could continue GI configuration.
So to identify the exact disk I used PROGRAM option. The above script looks through `/dev/dm-*` devices and if any of them satisfy the condition, for example:
then device name will be changed to /dev/oracleasm/asm-disk1, owner:group to grid:asmadmin and permission to 0660
But on my new servers same udev rule was not working anymore. (Of course, it needs more investigation, but our time is really valuable and never enough and if we know another solution that works and is acceptable- let’s just use it)
Solution:
I used udevadm command to identify other properties of these devices and wrote new udev rule (to see all properties, just remove grep):
# udevadm info --query=property --name /dev/mapper/asm1 | grep DM_UUID
DM_UUID=mpath-360050768028200a9a40000000000001c
sshd and chronyd services on the database server were in a failed state and not able to start because of the permission problem on their configuration files. Permissions on these files were correct and services should have been able to start, so there was something else… let’s dig into the details.
# systemctl status sshd
â sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2019-07-09 12:21:49 UTC; 32s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 124026 ExecStart=/usr/sbin/sshd -D $OPTIONS (code=exited, status=1/FAILURE)
Main PID: 124026 (code=exited, status=1/FAILURE)
Jul 09 12:21:49 node03 systemd[1]: Failed to start OpenSSH server daemon.
Jul 09 12:21:49 node03 systemd[1]: Unit sshd.service entered failed state.
Jul 09 12:21:49 node03 systemd[1]: sshd.service failed
`journalctl -xe` shows:
-- Unit sshd.service has begun starting up.
Jul 09 12:26:03 node03 sshd[129121]: /etc/ssh/sshd_config: Permission denied
Jul 09 12:26:03 node03 systemd[1]: sshd.service: main process exited, code=exited, status=1/FAILURE
Jul 09 12:26:03 node03 systemd[1]: Failed to start OpenSSH server daemon.
-- Subject: Unit sshd.service has failed
The same problem was happening with chronyd service. It was claiming about /etc/chrony.conf file. Incorrect time on database servers can cause node evictions.
Reason:
If permissions on these files are correct, we can think about SELinux, let’s check:
# getenforce
Enforcing
Solution:
Disable SELinux and reboot the server:
# vim /etc/selinux/config
SELINUX=disabled
# reboot
Summary:
I consider SELinux as a non-desirable service on the database servers. But I appreciate opinion of my colleages/friends and I want to share it with you.
SELinux can be enabled with the correct config in RHEL 4,5,6 – “Starting with Oracle Database 11g Release 2 (11.2), the Security Enhanced Linux (SELinux) feature is supported for Oracle Linux 4, Oracle Linux 5, Oracle Linux 6, Red Hat Enterprise Linux 4, Red Hat Enterprise Linux 5, and Red Hat Enterprise Linux 6. https://docs.oracle.com/cd/E11882_01/install.112/e47689/pre_install.htm#LADBI1092”
“SELinux is a good security tool and usually I only disable it as a last resort or if the software doesn’t support it. “
There are a lot of warnings “kernel: serial8250: too much work for irq4 ” in /var/log/messages and are likely your system experiences stability problems. And can lead to Oracle cluster node evictions.
Cause:
The problem was related to Azure OAM Agent pushing very large messages to serial console. The problem was introduced by the latest update of the Azure OMS agent.
Temporary Solution:
Temporarily remove OMS Linux Agent Extension until Microsoft resolves this bug:
1. On Azure portal click the link of the affected VM. 2. Click the “Extensions” section. 3. Click the OMS Linux Agent in the list. 4. Click the “Uninstall” button at the top
When you make sure that OMS agent bug is fixed (should be verified with Microsoft support), then you can reinstall the pluggin.
I am not able to receive email alerts from database server. Because message transfer agent is trying to connect to the Google SMTP via IPv6,which fails.
# tail /var/log/maillog
Jun 12 15:35:10 rac1 postfix/smtp[19725]:connect to
gmail-smtp-in.l.google.com [2607:f8b0:400c:c0b::1a]:25:
Network is unreachable
Solution:
Configure Postfix not to use IPv6 by editing /etc/postfix/main.cf with the following:
In our database there is turned on auditing on some operations and audit records go to OS.
SYS> show parameter audit_file_dest
NAME TYPE VALUE
------------------ ----------- ------------------------------
audit_file_dest string /u01_log/audit/orcl
SYS > show parameter audit_trail
NAME TYPE VALUE
------------- ----------- -----------
audit_trail string OS
Our security administrators are using SIEM to monitor suspicious activities and they want database to send audit records to this third party tool.
I thought that I could somehow indicate directory “/u01_log/audit/orcl” from where *.aud files would be uploaded to SIEM, but I was wrong. Some tools may be able to use these *.aud files but not SIEM and let’s configure our database to be able to send audit records to it.
1. Connect to a database instance as sysdba user
SQL> connect / as sysdba
2. Set audit trail to OS
SQL> alter system set audit_trail=OS;
3. Enable auditing for system users if you need to audit activities of sys user(optional)
SQL> alter system set audit_sys_operations=TRUE;
4. Set rsyslog facility and severity(needs database restart)
SQL> alter system set audit_syslog_level=local5.info scope=spfile sid='*';
Sometimes(it’s my third time) I have problems with windows vpn, getting error 720.
It has a “simple”(when you know(:) solution but every time I come across this problem I forget how I solved it before and decided to write solution here.
In Device Manager under “Network adapters” , there is yellow exclamation mark over “WAN Miniport (IP)” and “WAN Miniport(Network Monitor)”
Let’s uninstall it.
Uninstalling these miniports are not allowed. So first of all you should change it with different driver.
Locate your mouse on these driver and right click -> Update Driver Software->Browse my computer for driver software->Let me pick from a list of …-> uncheck “Show compatible hardware” and then choose for example Philips ->
When instead of WAN Miniport (IP) there appears the above driver , you can uninstall it.
Do the same steps for Network Monitor driver.
Now we do not have these drivers with exclamation marks.. because we do not have them at all 🙂
Now we are going to install these drivers with devcon.exe utility.
This post is dedicated to the Oracle Enterprise Linux 6.x installation on HP server with Dynamic Smart Array B320i Controller.
Brief description of the problem:
During OEL6 installation on HP server installer was not able to see local disks, on which system should be installed. Local disk specification was the following: Two 300GB disks involved into the RAID 1 and controller for RAID was Dynamic Smart Array B320i.
Installer was able to see multipath devices and any other external devices, like flash drive, but not local disks.
On HP site, there is written that the minimum supported Oracle Linux updates for HP DL380e series are the following:
“Minimum support includes all future updates of the indicated release unless a maximum is listed. (Example: Oracle 6 implies support for Oracle 6.x, unless it’s specifically called out in the notes as "not supported with 6.x")”
And HP also declaims that it gives its drivers to the supported OSs manufacturers.
To tell the truth, I’ve tried Oracle Enterprise Linux 5.5, 6.0, 6.2 and 6.4 versions, but none of them was able to see the local storage.
After a lot of troubleshooting, I found the solution.
Solution:
The solution implies the HP Dynamic Smart Array SATA RAID Controller Driver for Red Hat Enterprise Linux 6 installation before OS installation.
1. Download drivers from the following link, click here.
Note: hpvsa-1.2.4-4.rhel6u1.x86_64.dd.gz is for RHEL6.1 hpvsa-1.2.4-4.rhel6u2.x86_64.dd.gz is for RHEL6.2 hpvsa-1.2.4-4.rhel6u3.x86_64.dd.gz is for RHEL6.3
Because of we are installing Oracle Enterprise Linux 6.2 which is based on RHEL6.2, we need hpvsa-1.2.4-4.rhel6u2.x86_64.dd.gz.
2. Extract gz file, you should have file with the extension of dd. Write this file to the flash drive and connect it to the server.
3. Put Oracle Enterprise Linux 6.2 installation disk and when the following window appears, press the TAB.
At the end of the command write install dd(separated from the previous command with space) and press Enter.
4. Choose the driver file, that we mentioned earlier and continue the installation.
Oracle EL should now see the local disk without any problem. Good Luck.
