sshd: /etc/ssh/sshd_config: Permission denied

Problem:

sshd and chronyd services on the database server were in a failed state and not able to start because of the permission problem on their configuration files. Permissions on these files were correct and services should have been able to start, so there was something else… let’s dig into the details.

# systemctl status sshd
 â sshd.service - OpenSSH server daemon
    Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
    Active: activating (auto-restart) (Result: exit-code) since Tue 2019-07-09 12:21:49 UTC; 32s ago
      Docs: man:sshd(8)
            man:sshd_config(5)
   Process: 124026 ExecStart=/usr/sbin/sshd -D $OPTIONS (code=exited, status=1/FAILURE)
Main PID: 124026 (code=exited, status=1/FAILURE)
Jul 09 12:21:49 node03 systemd[1]: Failed to start OpenSSH server daemon.
Jul 09 12:21:49 node03 systemd[1]: Unit sshd.service entered failed state.
Jul 09 12:21:49 node03 systemd[1]: sshd.service failed

`journalctl -xe` shows:

-- Unit sshd.service has begun starting up.
Jul 09 12:26:03 node03 sshd[129121]: /etc/ssh/sshd_config: Permission denied
Jul 09 12:26:03 node03 systemd[1]: sshd.service: main process exited, code=exited, status=1/FAILURE
Jul 09 12:26:03 node03 systemd[1]: Failed to start OpenSSH server daemon.
-- Subject: Unit sshd.service has failed

The same problem was happening with chronyd service. It was claiming about /etc/chrony.conf file. Incorrect time on database servers can cause node evictions.

Reason:

If permissions on these files are correct, we can think about SELinux, let’s check:

# getenforce 
Enforcing

Solution:

Disable SELinux and reboot the server:

# vim /etc/selinux/config
SELINUX=disabled

# reboot

Summary:

I consider SELinux as a non-desirable service on the database servers. But I appreciate opinion of my colleages/friends and I want to share it with you.

SELinux can be enabled with the correct config in RHEL 4,5,6 – “Starting with Oracle Database 11g Release 2 (11.2), the Security Enhanced Linux (SELinux) feature is supported for Oracle Linux 4, Oracle Linux 5, Oracle Linux 6, Red Hat Enterprise Linux 4, Red Hat Enterprise Linux 5, and Red Hat Enterprise Linux 6.
https://docs.oracle.com/cd/E11882_01/install.112/e47689/pre_install.htm#LADBI1092

SELinux is a good security tool and usually I only disable it as a last resort or if the software doesn’t support it.

“kernel: serial8250: too much work for irq4” potential problem caused by Azure OMS Agent

Problem:

There are a lot of warnings “kernel: serial8250: too much work for irq4 ” in /var/log/messages and are likely your system experiences stability problems. And can lead to Oracle cluster node evictions.

Cause:

The problem was related to Azure OAM Agent pushing very large messages to serial console. The problem was introduced by the latest update of the Azure OMS agent.

Temporary Solution:

Temporarily remove OMS Linux Agent Extension until Microsoft resolves this bug:

1. On Azure portal click the link of the affected VM.
2. Click the “Extensions” section.
3. Click the OMS Linux Agent in the list.
4. Click the “Uninstall” button at the top

When you make sure that OMS agent bug is fixed (should be verified with Microsoft support), then you can reinstall the pluggin.

Postfix: connect to gmail-smtp-in.l.google.com [2607:f8b0:400c:c0b::1a]:25: Network is unreachable

Problem:

I am not able to receive email alerts from database server. Because message transfer agent is trying to connect to the Google SMTP via IPv6, which fails.

# tail /var/log/maillog

Jun 12 15:35:10 rac1 postfix/smtp[19725]:connect to 
gmail-smtp-in.l.google.com [2607:f8b0:400c:c0b::1a]:25: 
Network is unreachable

Solution:

Configure Postfix not to use IPv6 by editing /etc/postfix/main.cf with the following:

[root@rac1 ~]# cat /etc/postfix/main.cf | grep inet_protocols
inet_protocols = ipv4

Restart Postfix and check the status:

[root@rac1 ~]# systemctl restart postfix

[root@rac1 ~]# systemctl status  postfix
 ● postfix.service - Postfix Mail Transport Agent
    Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
    Active: active (running) since Thu 2019-06-13 10:20:48 UTC; 52s ago
   Process: 17431 ExecStop=/usr/sbin/postfix stop (code=exited, status=0/SUCCESS)
   Process: 17449 ExecStart=/usr/sbin/postfix start (code=exited, status=0/SUCCESS)
   Process: 17445 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS)
   Process: 17442 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
  Main PID: 17520 (master)
    Memory: 3.0M
    CGroup: /system.slice/postfix.service
            ├─17520 /usr/libexec/postfix/master -w
            ├─17521 pickup -l -t unix -u
            └─17522 qmgr -l -t unix -u
 Jun 13 10:20:48 rac1.example.com systemd[1]: Starting Postfix Mail Transport Agent…
 Jun 13 10:20:48 rac1.example.com postfix/postfix-script[17518]: starting the Postfix mail system
 Jun 13 10:20:48 rac1.example.com postfix/master[17520]: daemon started -- version 2.10.1, configuration /etc/postfix
 Jun 13 10:20:48 rac1.example.com systemd[1]: Started Postfix Mail Transport Agent

Multipath configuration on RHEL6

1. Check if you have already installed device-mapper-multipath rpm, if not then install it.

rpm -qa device-mapper-multipath

2. If /etc/multipath.conf file doesn’t exist, then copy it from /usr/share/doc/device-mapper-multipath-*

cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf

3. Find WWIDs that should be added to multipath configuration.

# scsi_id -g -u /dev/sdb
36001438009b044d90000900000780000

4. Edit the /etc/multipath.conf configuration file

defaults {
        user_friendly_names yes
        path_grouping_policy    failover
}

blacklist {
        wwid "*"
}

blacklist_exceptions {
        wwid "36001438009b044d90000900000780000"
}

multipaths {
        multipath {
                wwid                    "36001438009b044d90000900000780000"
                alias                   asm1
        }
}

5.  Add module to the Linux kernel:

modprobe dm-multipath

6. Start multipath service:

service multipathd start

7. If you have any syntax errors or any parameters that do not work in your Linux version, the following command will show:

multipath -d

8. Commit the configuration:

multipath -v2

9. The following command must find the paths , or you have a bad configuration in multipath.conf file:

multipath -ll

10. Make devices configured after a reboot:

chkconfig multipathd on

If you have made any mistakes in multipath.conf file then correct them and do  the following steps to make changes take affect :

1. edit the /etc/multipath.conf

2.  Reload the multipath service:

service multipathd reload

3.  Remove all unused multipath devices

multipath -F

4.  Check again that syntax is correct:

multipath –d

5.  Commit the changes:

multipath –v2

Note that, this configuration is very simple, but it is working also perfectly.

For more multipath options and more sophisticated configuration, see the following documentation.

Installation problem of OEL6 on HP ProLiant DL360e Gen8 with HP Dynamic Smart Array B320i Controller

This post is dedicated to the Oracle Enterprise Linux 6.x installation on HP server with Dynamic Smart Array B320i Controller.

Brief description of the problem:

During OEL6 installation on HP server installer was not able to see local disks, on which system should be installed. Local disk specification was the following: Two 300GB disks involved into the RAID 1 and controller for RAID was Dynamic Smart Array B320i.

Installer was able to see multipath devices and any other external devices, like flash drive, but not local disks.

On HP site, there is written that the minimum supported Oracle Linux updates for HP  DL380e series are the following:

DL380e Gen8

  • Oracle Linux/UEK 6.2
  • Oracle Linux/UEK 5.8

Minimum support includes all future updates of the indicated release unless a maximum is listed. (Example: Oracle 6 implies support for Oracle 6.x, unless it’s specifically called out in the notes as "not supported with 6.x")”

And HP also declaims that it gives its drivers to the supported OSs manufacturers.

To tell the truth, I’ve tried Oracle Enterprise Linux 5.5, 6.0, 6.2 and 6.4 versions, but none of them was able to see the local storage.

After a lot of troubleshooting, I found the solution.

Solution:

The solution implies the HP Dynamic Smart Array SATA RAID Controller Driver for Red Hat Enterprise Linux 6 installation before OS installation.

1. Download drivers from the following link, click here.

Note: hpvsa-1.2.4-4.rhel6u1.x86_64.dd.gz is for RHEL6.1
hpvsa-1.2.4-4.rhel6u2.x86_64.dd.gz is for RHEL6.2
hpvsa-1.2.4-4.rhel6u3.x86_64.dd.gz is for RHEL6.3

Because of we are installing Oracle Enterprise Linux 6.2 which is based on RHEL6.2, we need hpvsa-1.2.4-4.rhel6u2.x86_64.dd.gz.

2. Extract gz file, you should have file with the extension of dd. Write this file to the flash drive and connect it to the server.

3. Put Oracle Enterprise Linux 6.2 installation disk and when the following window appears, press the TAB.

image

At the end of the command write install dd(separated from the previous command with space) and press Enter.

4. Choose the driver file, that we mentioned earlier and continue the installation.

Oracle EL should now see the local disk without any problem. Good Luck.

Cannot retrieve repository metadata (repomd.xml) for repository: c5-media

Description of the Problem

I have newly installed Centos 5 and when I was trying to run Add/Remove Software I got the following error:

Unable To Retrieve Software Info, Package Manager

Also I tried installing package using yum, but showing the following:

Error: Cannot retrieve repository metadata (repomd.xml) for repository: c5-media.
Please verify its path and try again

Solution of the Problem

su –
vi /etc/yum.repos.d/CentOS-Media.repo

find the string “enabled=1”  to “enabled=0”.

That’s it. Just re-run Package Manager.