Forceful startup of CRS, when minority VMs are down

If a minority of database nodes are down because of cloud maintenance, those nodes may not be startable. If CRS is also down on the remaining working nodes, manual intervention is required.

Before proceeding, confirm that the cluster still has majority quorum.

Majority formula = TRUNC((number of database nodes + number of quorum nodes) / 2) + 1

The cluster can only be started when the majority of voting members are available. If the majority of database nodes or quorum nodes are down, the steps below will not work.

Use the following procedure on each database node where CRS fails to start.



Procedure 1: Restart CRS cleanly

1. Temporarily disable CRS autostart

crsctl disable crs

2. Stop any running CRS processes

crsctl stop crs -f

It is normal to see errors such as CRS-4639 or CRS-4000 when running this command. You can continue with the next steps.

3. Kill any remaining ohasd.bin reboot processes

ps -ef | grep "ohasd.bin reboot" | grep -v grep | awk '{print $2}' | xargs kill -9 > /dev/null 2>&1

4. [Only if using FlashGrid cluster] Stop flashgrid_wait service

flashgrid-node stop-waiting

Expected output may look similar to this:

pkill -USR1 -f flashgrid_wait ... OK

5. Restart the ohasd services

systemctl restart ohasd
systemctl restart oracle-ohasd

6. Monitor CRS startup

First, check whether the Clusterware daemons are running:

crsctl status res -t -init

If the Clusterware daemons started successfully, check the cluster resources:

crsctl status res -t

If CRS does not start automatically, start it manually:

crsctl start crs -wait

If startup hangs on ora.storage, check the ASM alert log (alert_+ASM?.log).

Look for errors such as: ORA-15042, ORA-15040

If these errors are present, cancel the CRS startup, skip step 7, and continue with Procedure 2 below.

7. Re-enable CRS autostart

crsctl enable crs

Procedure 2: If CRS still does not start

Use this procedure if CRS did not start successfully and some CRS resources remain failed.

Repeat the following steps on each database node where CRS still fails to start.

1. Stop any running CRS processes

crsctl stop crs -f

2. Kill any remaining ohasd.bin reboot processes

ps -ef | grep "ohasd.bin reboot" | grep -v grep | awk '{print $2}' | xargs kill -9 > /dev/null 2>&1

3. Restart the ohasd services

systemctl restart ohasd
systemctl restart oracle-ohasd

4. Start only HAS

crsctl start has

5. Start ASM in nomount mode

Connect as the Grid Infrastructure owner, for example grid:

su - grid
sqlplus / as sysasm

Then start ASM in nomount mode:

startup nomount;

6. Try to mount all ASM diskgroups

alter diskgroup all mount;

7. If mounting all diskgroups fails, mount them one by one using force

For example:

alter diskgroup GRID mount force;
alter diskgroup DATA mount force;

Sometimes ASM delays background operations after an unclean shutdown. In that case, you may see a message similar to this in alert_+ASM?.log:

WARNING: Background operations delayed until 08/08/23 21:22:21 because ASM was not stopped cleanly and there could be disconnected client(s)

Do not cancel the running command. Wait until the time shown in the message. The diskgroup should mount after that delay.

8. Re-enable CRS autostart

crsctl enable crs

9. Check cluster status

crsctl status res -t

Daylight saving time support in Oracle CRS

Dear readers,

I am glad to announce that my blog has been entered in Top 50 Oracle Blogs. For more information about Top 100 Oracle Blogs And Websites for Oracle DBAs To Follow in 2018 please visit https://blog.feedspot.com/oracle_blogs. You will improve your knowledge and experience by following them. 

In this post, I want to share my experience of how I solved the daylight saving time problem with Oracle CRS. With the default setup, in case timezone changes on your system, the client/application who connects to the database remotely(local/BEQ connections have correct timezone) will still have old timezone information and will enter wrong data.

Some countries,  that are not affected by daylight saving time are lucky and does not have to worry about it. But if your servers are not located in lucky countries then you must make CRS DTS aware.

During the GI installation, Oracle saves Timezone information in $CRS_HOME/crs/install/s_crsconfig_hostname_env.txt file, that makes TZ not to change for CRS even it is changed on OS level.

Please note that timezone can be changed for the database using srvctl:

srvctl setenv database -env 'TZ=time zone'

But I do not recommend to do that, because you must do the same everytime you create a new database.
Better to change TZ globally at CRS level.

In simple words just commenting out the TZ variable in $CRS_HOME/crs/install/s_crsconfig_hostname_env.txt and restarting the CRS on each node just one time is enough to do that, but let’s check it.

1.  List the current timezone settings:

[root@rac1 ~]# timedatectl status|grep zone
Time zone: UTC (UTC, +0000)
[root@rac2 ~]#  timedatectl status|grep zone
Time zone: UTC (UTC, +0000)

2. Change timezone at OS level:

[root@rac1 ~]# timedatectl set-timezone Europe/Bratislava
[root@rac2 ~]# timedatectl set-timezone Europe/Bratislava

3. Check local and scan connections:

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL> select to_char(sysdate,'HH24:MI:SS AM')  dbtime from dual;

DBTIME
-----------
18:50:05 PM     <<<<<<<<<<<<Correct , same as OS

[oracle@rac1 ~]$ sqlplus marik/123@ORCL

SQL> select to_char(sysdate,'HH24:MI:SS AM') dbtime from dual;

DBTIME
-----------
16:50:10 PM     <<<<<<<<<<<<Incorrect

4. Comment TZ in the config file:

[root@rac1 ~]# cat /u01/app/18.3.0/grid/crs/install/s_crsconfig_rac1_env.txt|grep TZ=
#   the appropriate time zone name. For example, TZ=America/New_York
#TZ=UTC

[root@rac2 ~]# cat /u01/app/18.3.0/grid/crs/install/s_crsconfig_rac2_env.txt|grep TZ=
#   the appropriate time zone name. For example, TZ=America/New_York
#TZ=UTC

5. Restart CRS on both nodes:

[root@rac1 ~]#  crsctl stop crs
[root@rac1 ~]#  crsctl start crs -wait
[root@rac2 ~]#  crsctl stop crs
[root@rac2 ~]#  crsctl start crs -wait

6. Change timezone on OS level several times and check local & scan connections:

[root@rac1 ~]# timedatectl set-timezone Africa/Conakry
[root@rac2 ~]# timedatectl set-timezone Africa/Conakry

Important: You need to reconnect to the database(so consider that sessions must be disconnected and reconnected again, old connections have old settings)

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL> Select to_char(sysdate,'HH24:MI:SS AM') dbtime from dual;

DBTIME
-----------
17:15:56 PM <<<<<<<<<<<<Correct


[oracle@rac1 ~]$ sqlplus marik/123@ORCL

SQL> Select to_char(sysdate,'HH24:MI:SS AM') dbtime from dual;

DBTIME
-----------
17:15:27 PM <<<<<<<<<<<<Correct

Change one more time:

[root@rac1 ~]# timedatectl set-timezone America/Aruba
[root@rac2 ~]# timedatectl set-timezone America/Aruba

Exit connections and reconnect:

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL> Select to_char(sysdate,'HH24:MI:SS AM') dbtime from dual;

DBTIME
-----------
13:17:47 PM <<<<<<<<<<<<Correct

[oracle@rac1 ~]$ sqlplus marik/123@ORCL

SQL> Select to_char(sysdate,'HH24:MI:SS AM') dbtime from dual;

DBTIME
-----------
13:17:31 PM <<<<<<<<<<<<Correct