Oracle Clusterware is restarting database “” | shut down instance “” of database “” | start up instance “” of database “”

When I used data guard broker to switchover primary database to standby database sometimes broker writes “Oracle Clusterware is restarting database”  … hangs and times out.

The problem was that database was not registered with srvctl.

srvctl add database -d db_unique_name -o oracle_home
srvctl add instance -d db_unique_name -i instance_name1 -n node_name1
srvctl add instance -d db_unique_name -i instance_name2 -n node_name2

instead of db_unique_name, oracle_home , instance_name1, node_name1,  instance_name2, node_name2 enter values according to your database. 

Blocking session, find root blockers and kill

 

By the following script you are able to find root cause of the locks. Script do not display BACKGROUND processes that are locking others (for example, LGWR, DBWR), because killing them causes database crush.

In front of the kill immediate statement there is written additional information such as : username[machine]:sql_id[prev_sql_id]:program and then comes kill immediate statement.

with lk as (select blocking_instance||'.'||blocking_session blocker, inst_id||'.'||sid waiter
 from gv$session where blocking_instance is not null and blocking_session is not null and username is not null)
 select lpad(' ',2*(level-1))||waiter lock_tree from
 (select * from lk
 union all
 select distinct 'root', blocker from lk
 where blocker not in (select waiter from lk))
 connect by prior waiter=blocker start with blocker='root';
--Generate SQLs to kill top-level blockers
set serverout on
 declare
 sess varchar2(20);
 sessinfo varchar2(100);
 begin
 for i in
 (with lk as (select blocking_instance||'.'||blocking_session blocker, inst_id||'.'||sid waiter
 from gv$session where blocking_instance is not null and blocking_session is not null and username is not null)
 select distinct blocker from lk where blocker not in (select waiter from lk)
 )
 loop
 begin
 select regexp_substr(i.blocker,'[0-9]+$')||','||serial# ||',@' || regexp_substr(i.blocker,'[0-9]+'),
 substr(username||'['||machine||']:'||sql_id||'['||prev_sql_id||']:'||program,1,100) into sess, sessinfo
 from gv$session where inst_id = regexp_substr(i.blocker,'[0-9]+') and sid = regexp_substr(i.blocker,'[0-9]+$') and type='USER';
 dbms_output.put_line(sessinfo || ' ' || 'alter system kill session ''' || sess || ''' immediate;');
 exception when no_data_found
 then continue;
 end;
 end loop;
 end;
 /

DDL Trigger to audit schema changes

  1. Create sequence for assigning numbers to the events:
    -- Create sequence
    create sequence SYS.DSQ_DDLEVENTS
    minvalue 1
    maxvalue 9999999999999999999999999999
    start with 1
    increment by 1
    cache 100;
  2. Create ddl events table
    -- Create table
    create table SYS.DDL_EVENTS
    (
     eventid NUMBER,
     inst_id   NUMBER,
     eventdate DATE,
     oraloginuser VARCHAR2(100),
     oradictobjname VARCHAR2(100),
     oradictobjowner VARCHAR2(100),
     oradictobjtype VARCHAR2(100),
     orasysevent VARCHAR2(100),
     machine VARCHAR2(100),
     program VARCHAR2(100),
     osuser VARCHAR2(100)
    )
    tablespace USERS;
  3. Create table for saving SQL statements(this is necessary because triggered sql statements may have several lines)
    -- Create table
    create table SYS.DDL_EVENTS_SQL
    ( eventid NUMBER,
     sqlline NUMBER,
     sqltext VARCHAR2(4000)
    )
    tablespace USERS;
  4. Create DDL trigger(it doesn’t degrade performance at all, by my experience)
    CREATE OR REPLACE TRIGGER sys.audit_ddl_trg
    AFTER DDL ON DATABASE
    DECLARE
    l_eventId NUMBER;
     l_sqlText ORA_NAME_LIST_T;
    BEGIN
    if ORA_SYSEVENT!='TRUNCATE' then
    
     SELECT dsq_ddlEvents.NEXTVAL INTO l_eventId FROM SYS.DUAL;
    INSERT INTO ddl_events (eventId,
     inst_id,
     EVENTDATE,
     ORALOGINUSER,
     ORADICTOBJNAME,
     ORADICTOBJOWNER,
     ORADICTOBJTYPE,
     ORASYSEVENT,
     machine,
     program,
     osuser
     )
     ( SELECT l_eventId,
     inst_id,
     SYSDATE,
     ORA_LOGIN_USER,
     ORA_DICT_OBJ_NAME,
     ORA_DICT_OBJ_OWNER,
     ORA_DICT_OBJ_TYPE,
     ORA_SYSEVENT,
     machine,
     program,
     osuser
     FROM SYS.DUAL
     right outer join
     SYS.GV$SESSION s on (1=1)
     WHERE s.sid=SYS_CONTEXT('USERENV','SID')
     and SYS_CONTEXT('USERENV','INSTANCE')=s.inst_id);
     FOR l IN 1..ORA_SQL_TXT(l_sqlText) LOOP
     INSERT INTO ddl_events_sql
     ( eventId, sqlLine, sqlText )
     VALUES
     ( l_eventId, l, l_sqlText(l) );
     END LOOP;
     end if;
    
     exception when others then
     null;
    END;

Oracle: Audit DMLs by specific user

Ordinary auditing do not have option to indicate audit some activities done by specific user.

I mean, you cannot write the following:

audit insert on my_schema.my_table by my_user;  <<—-not possible. The right statement is:
audit insert on my_schema.my_table by access;
or
audit insert on my_schema.my_table by session;

If I want to audit only activities done by my_user, one of the way is to create audit policy like the following;

begin
dbms_fga.add_policy(
object_schema=>’my_schema‘,
object_name=> ‘my_table‘,
policy_name=> ‘my_policy’,
audit_condition => ‘sys_context(”USERENV”,”CURRENT_USER”)=”MY_USER”’,
enable => TRUE,
statement_types => ‘INSERT, UPDATE, DELETE’,
audit_column_opts => dbms_fga.all_columns);
END;

So audit_condition gives the opportunity to check something and in this case we are checking user that is running statements indicated in statement_types option.

–Logs will be located here

SELECT * FROM dba_fga_audit_trail

–To see what policies we have

SELECT * FROM dba_audit_policies

HOW TO CATALOG TAPE BACKUP PIECES

You can catalog TAPE backup piece only using automatic channel:

RMAN> configure channel device type 'SBT_TAPE' parms <mml parameters>

Example:

configure channel device type 'SBT_TAPE' parms='ENV=(NB_ORA_CLIENT=ClientHostName,NB_ORA_SERV=backupServerHostName)';
RMAN> catalog device type 'SBT_TAPE' backuppiece 'arch_dEYC_ub4qtfud9_s20836_p1_t903346601';

CURSOR_SHARING effect on database performance, latch: shared pool

I have upgraded our database from 11g to 12c and after that query performance degraded significantly.

I have generated ADDM report during the problematic period and found that the main problem was hard parses :

1  Hard Parse Due to Literal Usage           10.38 | 43.55%        1

Finding 1: Hard Parse Due to Literal Usage
Impact is 10.38 active sessions, 43.55% of total activity.
———————————————————-
SQL statements were not shared due to the usage of literals. This resulted in
additional hard parses which were consuming significant database time.

For us it is not new that our developers are not using bind variables so shared pool was/is growing and growing to retain all of the parsed SQLs and their execution plans.

But the same codes were running on 11g and were working fine. Just after upgrade database started to feel that bad 🙂

The reason is that 12c has major change in optimize behavior. So if bind variables are not used in existing application then you need to use CURSOR_SHARING=FORCE option, old value of this parameter was CURSOR_SHARING=EXACT.

alter system set cursor_sharing=FORCE;

After that database started to feel better but in any case I cleared shared pool(for clearing old,not necessary parses):

alter system flush shared_pool;

Database started to feel better!

Good Luck!

SESSIONS WAITING ON INACTIVE TRANSACTION BRANCH, GLOBAL HASH COLLISION 12c, 11g Oracle RAC(distributed transaction)

When performing XA transactions against a multi-node Oracle RAC configuration, some branches of the transaction may not commit.. this is a known bug, but to tell the truth no bug fix helped me to solve this problem until I came across IBM technote. https://www-304.ibm.com/support/docview.wss?uid=swg21460967

There are several workarounds but I prefer Work around 1.  I have used it and works perfectly.

1. If using pfile (init.ora) files, add the following line to the file:

_clusterwide_global_transactions=false

2. If using an spfile, issue the following command from SQL*Plus:

alter system set “_clusterwide_global_transactions”=false scope=spfile

3. Restart the database (you can restart nodes , one by one)

Problem should dissapear.

 

Upgrade Oracle Database from 11g to 12c

  1. Download Oracle 12c software fromhttp://www.oracle.com/technetwork/database/enterprise-edition/downloads/database12c-linux-download-1959253.html

    unzip files:

    unzip linuxamd64_12102_database_1of2.zip
    unzip linuxamd64_12102_database_2of2.zip
  2. Make another home for 12c.
    mkdir -p /u01/app/oracle/product/12.1.0/dblb
  3. Change permissions for /u01 directory.  If 11g home is also located in /u01 you have already done the steps bellow. Just check that permissions are the following:
    chown -R oracle:oinstall /u01
    chmod -R 775 /u01
  4. Change the following parameters in response file, other parameters just leave blank.
    Response file is located in installation directory… extract_drectory/database/response/db_install.rsp

     oracle.install.option=INSTALL_DB_SWONLY
     ORACLE_HOSTNAME=DBServerHostname
     UNIX_GROUP_NAME=oinstall
     INVENTORY_LOCATION=/u01/app/oraInventory
     SELECTED_LANGUAGES=en
     ORACLE_HOME=/u01/app/oracle/product/12.1.0/dblb
     ORACLE_BASE=/u01/app/oracle
     oracle.install.db.InstallEdition=EE
     oracle.install.db.DBA_GROUP=dba
     oracle.install.db.OPER_GROUP=dba
     oracle.install.db.BACKUPDBA_GROUP=dba
     oracle.install.db.DGDBA_GROUP=dba
     oracle.install.db.KMDBA_GROUP=dba
     SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
     DECLINE_SECURITY_UPDATES=true
  5. Go to the 12c installation folder  and run runInstaller, to install 12c home:
     ./runInstaller -silent -responseFile /install/database/response/db_install.rsp  -waitforcompletion -showProgress

    When it asks , connect to the server via root user and run

    /u01/app/oracle/product/12.1.0/dblb/root.sh
  6. At this time your database should be turned on from 11g home. Connect to the database via SYS user and run the following scripts: emremove.sql will remove EM repository. olspreupgrade.sql will run preupgrade scripts

    You should also purge the recyclebin. For reducing upgrade time.

    Note: these scripts should be run to the open database , that is turned on by 11g

    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dblb
     sqlplus / as sysdba
     @/u01/app/oracle/product/12.1.0/dblb/rdbms/admin/emremove.sql
     @/u01/app/oracle/product/12.1.0/dblb/rdbms/admin/olspreupgrade.sql
     purge recyclebin;
  7. Run DBUA from 12c home to upgrade existing database
    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dblb
    /u01/app/oracle/product/12.1.0/dblb/bin/dbua -silent \
    -sid ORCL \
    -oracleHome /u01/app/oracle/product/11.2.0/dblb \
    -diagnosticDest /u01/app/oracle \
    -recompile_invalid_objects true \
    -degree_of_parallelism 40 \
    -upgradeTimezone \
    -emConfiguration NONE \
    -keepHiddenParams \
    -gatheringStatistics \
    -upgrade_parallelism 40
  8. To check that everything was upgraded successfully, after successful message from the previous command, check the following:
    cat /etc/oratab
    ORCL:/u01/app/oracle/product/12.1.0/dblb:N
    cat /u01/app/oraInventory/ContentsXML/inventory.xml
     <VERSION_INFO>
     <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
     <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
     </VERSION_INFO>

    Update the following parameters in your  .bash_profile:

     export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dblb
     export LD_LIBRARY_PATH=/u01/app/oracle/product/12.1.0/dblb/lib
  9. Network file configuration. Stop listener that was previously started from 11g. Move listener.ora and tnsnames.ora files to 12c home and rename them in old location. Start the listener from 12c home.
    /u01/app/oracle/product/11.2.0/dblb/bin/lsnrctl stop
    cp /u01/app/oracle/product/11.2.0/dblb/network/admin/listener.ora /u01/app/oracle/product/12.1.0/dblb/network/admin
    cp /u01/app/oracle/product/11.2.0/dblb/network/admin/tnsnames.ora /u01/app/oracle/product/12.1.0/dblb/network/admin
    mv /u01/app/oracle/product/11.2.0/dblb/network/admin/tnsnames.ora /u01/app/oracle/product/11.2.0/dblb/network/admin/tnsnames.ora.old
    mv /u01/app/oracle/product/11.2.0/dblb/network/admin/listener.ora /u01/app/oracle/product/11.2.0/dblb/network/admin/listener.ora.old
    /u01/app/oracle/product/12.1.0/dblb/bin/lsnrctl start
  10. Connect to the database using 12c home and check again the version in v$instance view:
    export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dblb
    sqlplus / as sysdba
    select instance_name,version,status
    from v$instance;
  11. If you want to deinstall 11g home run the following :
    /u01/app/oracle/product/11.2.0/dblb/deinstall/deinstall