sshd: /etc/ssh/sshd_config: Permission denied

July 9, 2019 6 Comments

Problem:

sshd and chronyd services on the database server were in a failed state and not able to start because of the permission problem on their configuration files. Permissions on these files were correct and services should have been able to start, so there was something else… let’s dig into the details. 

# systemctl status sshd
 â sshd.service - OpenSSH server daemon
    Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
    Active: activating (auto-restart) (Result: exit-code) since Tue 2019-07-09 12:21:49 UTC; 32s ago
      Docs: man:sshd(8)
            man:sshd_config(5)
   Process: 124026 ExecStart=/usr/sbin/sshd -D $OPTIONS (code=exited, status=1/FAILURE)
Main PID: 124026 (code=exited, status=1/FAILURE)
Jul 09 12:21:49 node03 systemd[1]: Failed to start OpenSSH server daemon.
Jul 09 12:21:49 node03 systemd[1]: Unit sshd.service entered failed state.
Jul 09 12:21:49 node03 systemd[1]: sshd.service failed

`journalctl -xe` shows:

-- Unit sshd.service has begun starting up.
Jul 09 12:26:03 node03 sshd[129121]: /etc/ssh/sshd_config: Permission denied
Jul 09 12:26:03 node03 systemd[1]: sshd.service: main process exited, code=exited, status=1/FAILURE
Jul 09 12:26:03 node03 systemd[1]: Failed to start OpenSSH server daemon.
-- Subject: Unit sshd.service has failed

The same problem was happening with chronyd service. It was claiming about /etc/chrony.conf file. Incorrect time on database servers can cause node evictions.

Reason:

If permissions on these files are correct, we can think about SELinux, let’s check:

# getenforce 
Enforcing

Solution:

Disable SELinux and reboot the server:

# vim /etc/selinux/config
SELINUX=disabled

# reboot

Summary:

I consider SELinux as a non-desirable service on the database servers. But I appreciate opinion of my colleages/friends and I want to share it with you.

SELinux can be enabled with the correct config in RHEL 4,5,6 – “Starting with Oracle Database 11g Release 2 (11.2), the Security Enhanced Linux (SELinux) feature is supported for Oracle Linux 4, Oracle Linux 5, Oracle Linux 6, Red Hat Enterprise Linux 4, Red Hat Enterprise Linux 5, and Red Hat Enterprise Linux 6.
https://docs.oracle.com/cd/E11882_01/install.112/e47689/pre_install.htm#LADBI1092”

“SELinux is a good security tool and usually I only disable it as a last resort or if the software doesn’t support it. “

Filed under DB, Linux, Oracle Enterprise Linux, OS, Uncategorized Tagged with , ,

Postfix: connect to gmail-smtp-in.l.google.com [2607:f8b0:400c:c0b::1a]:25: Network is unreachable

June 13, 2019 Leave a comment

Problem:

I am not able to receive email alerts from database server. Because message transfer agent is trying to connect to the Google SMTP via IPv6, which fails.

# tail /var/log/maillog

Jun 12 15:35:10 rac1 postfix/smtp[19725]:connect to 
gmail-smtp-in.l.google.com [2607:f8b0:400c:c0b::1a]:25: 
Network is unreachable

Solution:

Configure Postfix not to use IPv6 by editing /etc/postfix/main.cf with the following:

[root@rac1 ~]# cat /etc/postfix/main.cf | grep inet_protocols
inet_protocols = ipv4

Restart Postfix and check the status:

[root@rac1 ~]# systemctl restart postfix

[root@rac1 ~]# systemctl status  postfix
 ● postfix.service - Postfix Mail Transport Agent
    Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
    Active: active (running) since Thu 2019-06-13 10:20:48 UTC; 52s ago
   Process: 17431 ExecStop=/usr/sbin/postfix stop (code=exited, status=0/SUCCESS)
   Process: 17449 ExecStart=/usr/sbin/postfix start (code=exited, status=0/SUCCESS)
   Process: 17445 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS)
   Process: 17442 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
  Main PID: 17520 (master)
    Memory: 3.0M
    CGroup: /system.slice/postfix.service
            ├─17520 /usr/libexec/postfix/master -w
            ├─17521 pickup -l -t unix -u
            └─17522 qmgr -l -t unix -u
 Jun 13 10:20:48 rac1.example.com systemd[1]: Starting Postfix Mail Transport Agent…
 Jun 13 10:20:48 rac1.example.com postfix/postfix-script[17518]: starting the Postfix mail system
 Jun 13 10:20:48 rac1.example.com postfix/master[17520]: daemon started -- version 2.10.1, configuration /etc/postfix
 Jun 13 10:20:48 rac1.example.com systemd[1]: Started Postfix Mail Transport Agent

Filed under Linux, Oracle Enterprise Linux, OS Tagged with , , ,

pam_systemd(sshd:session): Failed to create session: Failed to activate service ‘org.freedesktop.login1’: timed out

April 16, 2019 Leave a comment

Problem:

1. Slow ssh connections
2. System seems slow when trying to su to another user

/var/log/secure contains the following errors:

pam_systemd(sshd:session): Failed to create session: Failed to activate
service 'org.freedesktop.login1': timed out

Solution:

1. Restart systemd-logind service

# systemctl restart systemd-logind

2. Restart server

# reboot

Note that the mentioned solutions are considered as temporary solutions (Frankly, I’ve never seen this error after restart. The problem happened with our two customers, who changed sshd_config file and did “something” after that, so the problem was caused by humman error in my all cases), for more information about this problem please see article at redhat site
https://access.redhat.com/discussions/3536621 .

Filed under Linux Tagged with , , , ,

How to display certain line from a text file in Linux?

March 9, 2019 Leave a comment

Sometimes script fails and error mentiones the line number, where there is a mistake. One option is to open a file and go to the mentioned line.

I am going to show you how to use SED to print only the certain line from the script file:

# sed -n ’80p’ /u01/app/18.3.0/grid/crs/script/mount-dbfs.sh

“My 80th line”

Where 80 is the line number and p “print[s] the current pattern space”

Filed under Linux Tagged with ,

Check if a port on a remote system is reachable (without telnet)

February 4, 2019 2 Comments

Problem:

I wanted to check if our customer had an issue with network access. I asked him to run telnet command, but because of their security reasons, telnet rpm was not installed at all.

Solution:

One of the solution would be to run nc 🙂 but of course nc was not installed. 🙂

The following command helped me in that situation:

cat < /dev/tcp/192.168.3.4/3260

The port is open if there is no output, but if you receive the following, then the port is closed:

-bash: connect: Connection refused
-bash: /dev/tcp/192.168.3.4/3261: Connection refused

 

Filed under Linux Tagged with ,

Install blobfuse on RHEL7 fails

December 19, 2018 Leave a comment

Problem: 

Not able to install blobfuse on RHEL7.

Error:

Error: Package: blobfuse-1.0.2-1.x86_64 (packages-microsoft-com-prod)
Requires: libgnutls.so.26(GNUTLS_1_4)(64bit)
Error: Package: blobfuse-1.0.2-1.x86_64 (packages-microsoft-com-prod)
Requires: libgnutls.so.26(GNUTLS_2_10)(64bit)
Error: Package: blobfuse-1.0.2-1.x86_64 (packages-microsoft-com-prod)
Requires: libgnutls.so.26()(64bit)
You could try using –skip-broken to work around the problem
You could try running: rpm -Va –nofiles –nodigest

Solution:

sudo yum remove packages-microsoft-prod-1.0-1.el7.noarch

sudo yum clean all

sudo rm -rf /var/cache/yum

sudo rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm

sudo yum install blobfuse fuse -y

Filed under Linux Tagged with , ,

vncserver fails to start

December 18, 2018 Leave a comment

VNC is useful when you want to run DBCA or some other tool in GUI mode. If the connection between your computer and server fails, running application in VNC continues working and you can reconnect to your previous session.

I have noticed very strange behavior of VNC on my Linux 7.5, it maybe the same on other versions.

Problem:

Start VNC:

$ vncserver -geometry 1024×1024

New ‘stbyracq.example.com:1 (oracle)’ desktop is stbyracq.example.com:1

Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/stbyracq.example.com:1.log

Check VNC process:

$ ps -ef|grep vnc
root 20629 2570 0 16:36 pts/1 00:00:00 grep –color=auto vnc

Shows only GREP process.

Let’s check the log file:

$ cat /home/oracle/.vnc/stbyracq.example.com:1.log

Xvnc TigerVNC 1.8.0 – built Aug 31 2018 12:04:07
Copyright (C) 1999-2017 TigerVNC Team and many others (see README.txt)
See http://www.tigervnc.org for information on TigerVNC.
Underlying X server release 12001000, The X.Org Foundation

Tue Dec 18 16:38:05 2018
vncext: VNC extension running!
vncext: Listening for VNC connections on all interface(s), port 5901
vncext: created VNC server for screen 0
/usr/bin/xterm: cannot load font ‘-misc-fixed-medium-r-semicondensed–13-120-75-75-c-60-iso10646-1’
Killing Xvnc process ID 21100
XIO: fatal IO error 2 (No such file or directory) on X server “:1”
after 90 requests (90 known processed) with 4 events remaining.

Solution:

Check xstartup script:

$ cat /home/oracle/.vnc/xstartup
#!/bin/sh

unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
/etc/X11/xinit/xinitrc
vncserver -kill $DISPLAY

The line at the end is killing vncserver (which is trange), remove that line:

$ cat /home/oracle/.vnc/xstartup
#!/bin/sh

unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
/etc/X11/xinit/xinitrc

Start VNC:

$ vncserver -geometry 1024×1024

New ‘stbyracq.example.com:1 (oracle)’ desktop is stbyracq.example.com:1

Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/stbyracq.example.com:1.log

Check VNC process:

ps -ef|grep vnc
oracle 22690 1 0 16:40 pts/1 00:00:00 /bin/Xvnc :1 -auth /home/oracle/.Xauthority -desktop stbyracq.example.com:1 (oracle) -fp catalogue:/etc/X11/fontpath.d -geometry 1024×1024 -pn -rfbauth /home/oracle/.vnc/passwd -rfbport 5901 -rfbwait 30000
oracle 23106 20841 0 16:41 pts/1 00:00:00 grep –color=auto vnc

Now you can connect to VNC server.

 

Filed under Linux Tagged with , , ,

← Older posts