Tuesday 10 August 2021

DGMGRL - Switchover primary to standby database using dataguard

 Switchover primary to standby database using dataguard

Primary database : ZABBIX 

hostname: srv3.example.com

version :  12.1.0.2.0

Standby  Database - ZABBIXDR 

hostname: srv4.example.com

version :  12.1.0.2.0

  • Connect primary Database  and check both primary and dr configuration status:

 [oracle@srv3 ~]$ dgmgrl sys/system123@zabbix

DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production


Copyright (c) 2000, 2013, Oracle. All rights reserved.


Welcome to DGMGRL, type "help" for information.

Connected as SYSDBA.

DGMGRL> show configuration;


Configuration - zabbix


  Protection Mode: MaxPerformance

  Members:

  zabbix   - Primary database

    zabbixdr - Physical standby database


Fast-Start Failover: DISABLED


Configuration Status:

SUCCESS   (status updated 9 seconds ago)


DGMGRL> show database zabbix


Database - zabbix


  Role:               PRIMARY

  Intended State:     TRANSPORT-ON

  Instance(s):

    ZABBIX


Database Status:

SUCCESS


DGMGRL> show database zabbixdr


Database - zabbixdr


  Role:               PHYSICAL STANDBY

  Intended State:     APPLY-ON

  Transport Lag:      0 seconds (computed 0 seconds ago)

  Apply Lag:          0 seconds (computed 0 seconds ago)

  Average Apply Rate: 2.00 KByte/s

  Real Time Query:    OFF

  Instance(s):

    ZABBIXDR


Database Status:

SUCCESS


DGMGRL>


  • Create restore point before switchover on both primary and standby database. Used in case of failure. 
  • [oracle@srv3 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 12.1.0.2.0 Production on Mon Jul 19 17:30:32 2021


    Copyright (c) 1982, 2014, Oracle.  All rights reserved.


    Connected to:

    Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

    With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options


    SQL> set lines 200

    SQL> select name, instance_name,db_unique_name, open_mode, database_role, flashback_on  current_scn,to_char(standby_became_primary_scn) failover_scn from v$database,v$instance;


    NAME      INSTANCE_NAME    DB_UNIQUE_NAME                 OPEN_MODE            DATABASE_ROLE    CURRENT_SCN        FAILOVER_SCN

    --------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

    ZABBIX    ZABBIX           ZABBIX                         READ WRITE           PRIMARY          NO                 0


    SQL>


    SQL> CREATE RESTORE POINT before_switchover GUARANTEE FLASHBACK DATABASE;


    Restore point created.


    SQL>

  • Create restore point before switchover on  standby database. Used in case of failure. 
  •    

    [oracle@srv4 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 12.1.0.2.0 Production on Mon Jul 19 17:31:49 2021


    Copyright (c) 1982, 2014, Oracle.  All rights reserved.


    Connected to:

    Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

    With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options


    SQL> set lines 200

    SQL> select name, instance_name,db_unique_name, open_mode, database_role, flashback_on  current_scn,to_char(standby_became_primary_scn) failover_scn from v$database,v$instance;


    NAME      INSTANCE_NAME    DB_UNIQUE_NAME                 OPEN_MODE            DATABASE_ROLE    CURRENT_SCN        FAILOVER_SCN

    --------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

    ZABBIX    ZABBIXDR         ZABBIXDR                       MOUNTED              PHYSICAL STANDBY NO                 0


    SQL> CREATE RESTORE POINT before_switchover GUARANTEE FLASHBACK DATABASE;

    CREATE RESTORE POINT before_switchover GUARANTEE FLASHBACK DATABASE

    *

    ERROR at line 1:

    ORA-38784: Cannot create restore point 'BEFORE_SWITCHOVER'.

    ORA-01153: an incompatible media recovery is active

    SQL>

  • NOTE : Disable  redo apply on data guard om dgmrl console . Create restore point on standby and apply on again .
  • DGMGRL> EDIT DATABASE zabbixdr SET STATE=APPLY-OFF;

    Succeeded.

    DGMGRL>


    SQL>  CREATE RESTORE POINT before_switchover GUARANTEE FLASHBACK DATABASE;


    Restore point created.


    SQL>


    DGMGRL> EDIT DATABASE zabbixdr SET STATE=APPLY-ON;

    Succeeded.


    DGMGRL> show database zabbixdr


    Database - zabbixdr


      Role:               PHYSICAL STANDBY

      Intended State:     APPLY-ON

      Transport Lag:      0 seconds (computed 0 seconds ago)

      Apply Lag:          0 seconds (computed 0 seconds ago)

      Average Apply Rate: 2.00 KByte/s

      Real Time Query:    OFF

      Instance(s):

        ZABBIXDR


    Database Status:

    SUCCESS


    DGMGRL>

  • Login primary database and check status before switchover 
  •  

    SQL> SELECT CLIENT_PROCESS,PROCESS,SEQUENCE#,STATUS FROM V$MANAGED_STANDBY;


    CLIENT_P PROCESS    SEQUENCE# STATUS

    -------- --------- ---------- ------------

    ARCH     ARCH              54 OPENING

    ARCH     ARCH               0 CONNECTED

    ARCH     ARCH              53 CLOSING

    ARCH     ARCH              55 CLOSING

    LNS      LNS               56 WRITING


    SQL>

  • Login standby database and check status before switchover 
  •  

    SQL> SELECT STATUS FROM V$MANAGED_STANDBY WHERE PROCESS LIKE 'MRP%';


    STATUS

    ------------

    APPLYING_LOG


    SQL>

  • After checking the status. login to dgmgrl console and perform switchover.
  • DGMGRL> SWITCHOVER TO zabbixdr;

    Performing switchover NOW, please wait...

    New primary database "zabbixdr" is opening...

    Operation requires start up of instance "ZABBIX" on database "zabbix"


    Unable to connect to database using (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=srv3.example.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ZABBIX_DGMGRL)(INSTANCE_NAME=ZABBIX)(SERVER=DEDICATED)))

    ORA-12514: TNS:listener does not currently know of service requested in connect descriptor


    Failed.

    Warning: You are no longer connected to ORACLE.


    Please complete the following steps to finish switchover:

            start up and mount instance "ZABBIX" of database "zabbix"


    DGMGRL> DGMGRL>

  • Start new DR database srv3 server  in mount stage  
  •  

    SQL> startup mount;

    ORACLE instance started.


    Total System Global Area 1795162112 bytes

    Fixed Size                  2925456 bytes

    Variable Size             620760176 bytes

    Database Buffers         1157627904 bytes

    Redo Buffers               13848576 bytes

    Database mounted.

    SQL> SELECT CLIENT_PROCESS,PROCESS,SEQUENCE#,STATUS FROM V$MANAGED_STANDBY;


    CLIENT_P PROCESS    SEQUENCE# STATUS

    -------- --------- ---------- ------------

    ARCH     ARCH               0 CONNECTED

    ARCH     ARCH              59 CLOSING

    ARCH     ARCH               0 CONNECTED

    ARCH     ARCH               0 CONNECTED

    UNKNOWN  RFS                0 IDLE

    LGWR     RFS               60 IDLE

    UNKNOWN  RFS                0 IDLE

    N/A      MRP0              60 APPLYING_LOG


    8 rows selected.


    SQL>

  • Check the status of new primary database after switchover
  •  

    DGMGRL> show configuration;


    Configuration - zabbix


      Protection Mode: MaxPerformance

      Members:

      zabbixdr - Primary database

        zabbix   - Physical standby database


    Fast-Start Failover: DISABLED


    Configuration Status:

    SUCCESS   (status updated 0 seconds ago)


    DGMGRL> SHOW DATABASE zabbixdr;


    Database - zabbixdr


      Role:               PRIMARY

      Intended State:     TRANSPORT-ON

      Instance(s):

        ZABBIXDR


    Database Status:

    SUCCESS


    DGMGRL> SHOW DATABASE zabbix;


    Database - zabbix


      Role:               PHYSICAL STANDBY

      Intended State:     APPLY-ON

      Transport Lag:      0 seconds (computed 0 seconds ago)

      Apply Lag:          0 seconds (computed 0 seconds ago)

      Average Apply Rate: 107.00 KByte/s

      Real Time Query:    OFF

      Instance(s):

        ZABBIX


    Database Status:

    SUCCESS


    DGMGRL>


    Friday 9 July 2021

    patch - Combo of OJVM Component 12.1.0.2.181016 DB PSU + DB PSU 12.1.0.2.181016 12c on both primary and DR database

    Patch 28689146 - Combo of OJVM Component 12.1.0.2.181016 DB PSU + DB PSU 12.1.0.2.181016

    Patch 28259833 - Database Patch Set Update 12.1.0.2.181016

    Patch 28440711 - Oracle JavaVM Component 12.1.0.2.181016 Database PSU

    PSU and OJVM Patch on primary and Physical Standby database 

    Primary Database name : ZABBIX ( hostname: srv3.example.com)

    DR Database  Name : ZABBIXDR ( hostname: srv4.example.com)

    • STEP 1
    OPatch Utility

    use the OPatch utility version 12.2.0.1.13 or later to apply this patch. download from My Oracle Support patch 6880880 by selecting the 12.2.0.1.0 release.


    copy  patch 6880880  to  $ORACLE_HOME directory  
    unzip p6880880

    mv OPatch OPatch_old

    unzip p6880880 patch 

    • STEP 2
     

    SQL> select name,open_mode,database_role from v$database;

    SQL> select name,open_mode,database_role from v$database; NAME OPEN_MODE DATABASE_ROLE --------- -------------------- ---------------- ZABBIX MOUNTED PHYSICAL STANDBY SQL> alter database recover managed standby database cancel; Database altered. SQL> shut immediate; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> exit [oracle@srv4 ~]$ lsrnctl stop listener

    Take binary backup of DR database oracle home

     tar -czvf 12.1.0.tar.gz ./12.1.0

    tar -czvf oraInventory.tar.gz ./oraInventory

    •  STEP 3
    export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
    export PATH=$ORACLE_HOME/OPatch:$PATH:$ORACLE_HOME/bin
    export PATH=$ORACLE_HOME/perl/bin:$PATH

    Set oracle home path and after unziping patch check DB patch conflict status 

    Determine whether any currently installed interim patches conflict with the patch being installed, 28259833


    [oracle@srv4 28259833]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    PREREQ session
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-05_19-52-11PM_1.log
    
    Invoking prereq "checkconflictagainstohwithdetail"
    
    Prereq "checkConflictAgainstOHWithDetail" passed.
    
    OPatch succeeded.
    [oracle@srv4 28259833]$
    
    check conflict standby 
    [oracle@srv4 28259833]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    PREREQ session
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-05_19-52-11PM_1.log
    
    Invoking prereq "checkconflictagainstohwithdetail"
    
    Prereq "checkConflictAgainstOHWithDetail" passed.
    
    OPatch succeeded.
    
    • STEP 4  (Patch 28259833 - Database Patch Set Update 12.1.0.2.181016)

    • Set your current directory to the directory where the patch is located and then run the OPatch utility by entering the following commands: 

     
    • opatch apply -analyze (analyze patch standby) 
    [oracle@srv4 28259833]$ opatch apply -analyze
    • opatch apply 
    [oracle@srv4 28259833]$ opatch apply Oracle Interim Patch Installer version 12.2.0.1.21 Copyright (c) 2021, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.21 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-05_19-54-42PM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.21 Copyright (c) 2021, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.21 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-05_19-55-24PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 19769480 20299023 20831110 21359755 21948354 22291127 23054246 24006101 24732082 25171037 25755742 26609783 26713565 26925311 27338041 27547329 28259833 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying sub-patch '19769480' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.rdbms.deconfig, 12.1.0.2.0... Patching component oracle.xdk, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms.util, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.xdk.parser.java, 12.1.0.2.0... Patching component oracle.oraolap, 12.1.0.2.0... Patching component oracle.xdk.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.ldap.rsf.ic, 12.1.0.2.0... Applying sub-patch '20299023' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms.deconfig, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.rdbms.rsf.ic, 12.1.0.2.0... Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.ldap.rsf.ic, 12.1.0.2.0... Applying sub-patch '20831110' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.oraolap.dbscripts, 12.1.0.2.0... Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Applying sub-patch '21359755' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.assistants.server, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Applying sub-patch '21948354' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.rdbms.deconfig, 12.1.0.2.0... Patching component oracle.xdk, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.xdk.parser.java, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.xdk.rsf, 12.1.0.2.0... Applying sub-patch '22291127' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.ldap.client, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.oraolap, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.oraolap.dbscripts, 12.1.0.2.0... Applying sub-patch '23054246' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.rdbms.dv, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.install.deinstalltool, 12.1.0.2.0... Patching component oracle.ldap.rsf.ic, 12.1.0.2.0... Patching component oracle.oracore.rsf, 12.1.0.2.0... Patching component oracle.ctx, 12.1.0.2.0... Patching component oracle.xdk, 12.1.0.2.0... Patching component oracle.nlsrtl.rsf, 12.1.0.2.0... Patching component oracle.xdk.parser.java, 12.1.0.2.0... Patching component oracle.ctx.atg, 12.1.0.2.0... Applying sub-patch '24006101' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.sqlplus, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.network.listener, 12.1.0.2.0... Patching component oracle.network.rsf, 12.1.0.2.0... Patching component oracle.rdbms.dv, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.sqlplus.ic, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Applying sub-patch '24732082' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.rdbms.install.plugins, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Applying sub-patch '25171037' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.sqlplus.ic, 12.1.0.2.0... Patching component oracle.sqlplus, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.ordim.client, 12.1.0.2.0... Patching component oracle.ordim.client, 12.1.0.2.0... Patching component oracle.ordim.jai, 12.1.0.2.0... Patching component oracle.ordim.server, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms.install.plugins, 12.1.0.2.0... Patching component oracle.javavm.containers, 12.1.0.2.0... Applying sub-patch '25755742' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ] , [ oracle.has.crs, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.network.rsf, 12.1.0.2.0... Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.ldap.client, 12.1.0.2.0... Patching component oracle.oracore.rsf, 12.1.0.2.0... Patching component oracle.rdbms.deconfig, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.network.listener, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Applying sub-patch '26609783' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.oracore.rsf, 12.1.0.2.0... Applying sub-patch '26713565' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ] , [ oracle.has.crs, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.ldap.client, 12.1.0.2.0... Patching component oracle.rdbms.crs, 12.1.0.2.0... Patching component oracle.rdbms.deconfig, 12.1.0.2.0... Patching component oracle.xdk, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.xdk.parser.java, 12.1.0.2.0... Patching component oracle.xdk.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Applying sub-patch '26925311' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.network.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Applying sub-patch '27338041' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' ApplySession: Optional component(s) [ oracle.assistants.asm, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.network.rsf, 12.1.0.2.0... Patching component oracle.assistants.server, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.sqlplus.rsf, 12.1.0.2.0... Applying sub-patch '27547329' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' Patching component oracle.network.rsf, 12.1.0.2.0... Patching component oracle.tfa, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.ldap.rsf, 12.1.0.2.0... Patching component oracle.ctx.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.oracore.rsf, 12.1.0.2.0... Patching component oracle.ctx, 12.1.0.2.0... Patching component oracle.nlsrtl.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Applying sub-patch '28259833' to OH '/u01/app/oracle/product/12.1.0/dbhome_1' ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.oracore.rsf, 12.1.0.2.0... Patching component oracle.rdbms, 12.1.0.2.0... Patching component oracle.rdbms.dbscripts, 12.1.0.2.0... Patching component oracle.rdbms.rsf, 12.1.0.2.0... Patching component oracle.rdbms.rman, 12.1.0.2.0... Patching component oracle.rdbms.crs, 12.1.0.2.0... OPatch found the word "error" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: chmod: changing permissions of `/u01/app/oracle/product/12.1.0/dbhome_1/bin/extjobO': Operation not permitted make: [iextjob] Error 1 (ignored) Composite patch 28259833 successfully applied. OPatch Session completed with warnings. Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-05_19-55-24PM_1.log OPatch completed with warnings. [oracle@srv4 28259833]$ [oracle@srv4 28259833]$
    • STEP 5
    Check optach status by using ./opatch lsinventory 
     
    [oracle@srv4 OPatch]$ ./opatch lsinventory
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-05_20-09-20PM_1.log
    
    Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2021-07-05_20-09-20PM.txt
    --------------------------------------------------------------------------------
    Local Machine Information::
    Hostname: srv4.example.com
    ARU platform id: 226
    ARU platform description:: Linux x86-64
    
    Installed Top-level Products (1):
    
    Oracle Database 12c                                                  12.1.0.2.0
    There are 1 products installed in this Oracle Home.
    
    
    Interim patches (1) :
    
    Patch  28259833     : applied on Mon Jul 05 20:02:28 IST 2021
    Unique Patch ID:  22488632
    Patch description:  "Database Patch Set Update : 12.1.0.2.181016 (28259833)"
       Created on 8 Oct 2018, 06:11:15 hrs PST8PDT
    Sub-patch  27547329; "Database Patch Set Update : 12.1.0.2.180717 (27547329)"
    Sub-patch  27338041; "Database Patch Set Update : 12.1.0.2.180417 (27338041)"
    Sub-patch  26925311; "Database Patch Set Update : 12.1.0.2.180116 (26925311)"
    Sub-patch  26713565; "Database Patch Set Update : 12.1.0.2.171017 (26713565)"
    Sub-patch  26609783; "Database Patch Set Update : 12.1.0.2.170814 (26609783)"
    Sub-patch  25755742; "Database Patch Set Update : 12.1.0.2.170718 (25755742)"
    Sub-patch  25171037; "Database Patch Set Update : 12.1.0.2.170418 (25171037)"
    Sub-patch  24732082; "Database Patch Set Update : 12.1.0.2.170117 (24732082)"
    Sub-patch  24006101; "Database Patch Set Update : 12.1.0.2.161018 (24006101)"
    Sub-patch  23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)"
    Sub-patch  22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)"
    Sub-patch  21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
    Sub-patch  21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
    Sub-patch  20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
    Sub-patch  20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
    Sub-patch  19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
       Bugs fixed:
         19309466, 19902195, 18250893, 25437699, 19383839, 16756406, 18456643
         26546664, 22364044, 18845653, 19915271, 20172151, 18417036, 23713236
         24796092, 23140259, 19243521, 19658708, 18272672, 21153266, 19174430
         22243719, 20688221, 20493163, 21387964, 13542050, 22734547, 21623164
         19012119, 19932634, 19869255, 22232606, 18681056, 23324000, 25427662
         22068305, 24589081, 19439759, 19303936, 22916353, 24835538, 22353346
         21106027, 26444887, 23088803, 22529728, 26256131, 19134173, 20447445
         21188584, 19390567, 26513709, 25780343, 19769480, 21097043, 21225209
         26245237, 20677396, 19284031, 19450314, 19016730, 20919320, 22075064
         22551446, 22721409, 18440095, 22496904, 16439813, 18354830, 20596234
         22022760, 20936905, 23197103, 21514877, 26111842, 18990023, 22492533
         20173897, 24624166, 17210525, 21260431, 20181030, 25056052, 19370504
         21868720, 23068169, 19124589, 19402853, 19888853, 24341675, 17722075
         20882568, 25653109, 23026585, 18604692, 20717081, 25546608, 27370965
         19081128, 22173980, 23514710, 19178851, 20951038, 22168163, 25161298
         20569094, 24308635, 19791377, 19050649, 20920911, 19189525, 19469538
         27052607, 20598042, 22458049, 18988834, 23302839, 25307368, 17409174
         22729345, 22842151, 19238590, 16941434, 20387265, 24397438, 20673810
         23108128, 20356733, 22380919, 18436647, 23065323, 20825533, 19124336
         22294260, 24790914, 20284155, 25539063, 17365043, 25914276, 20952966
         22961508, 19176223, 21300341, 23237313, 18288842, 22353199, 22083366
         25670786, 21419850, 26898563, 19577410, 23294548, 24737064, 19931709
         25423453, 25547060, 23533807, 27726780, 24600330, 25600421, 18122373
         20043616, 23124895, 18856999, 21450666, 18893947, 20076781, 26633558
         26029780, 21196809, 21354456, 23725036, 20464614, 19562381, 24808595
         27375542, 19189317, 25669791, 18307021, 21917884, 19708632, 27213224
         25633101, 20711718, 18973548, 25982666, 19718981, 22826718, 25655390
         23567857, 21773465, 20250147, 19197175, 26263721, 19597439, 21387128
         22007324, 19180770, 19879746, 21785691, 20424183, 24285405, 26544823
         20322560, 22228324, 23172924, 22520320, 21575362, 25058080, 22365117
         22645009, 25165496, 18774543, 20124446, 21429602, 26153977, 19371175
         21863727, 18940497, 19074147, 22923409, 25489342, 21380789, 19154375
         19044962, 19532017, 19662635, 22374754, 20560611, 25654936, 21492036
         18705806, 19578247, 22024071, 22238921, 22809871, 21184223, 23089357
         19404068, 18921743, 19065677, 19018447, 19018206, 18308268, 19777862
         22223463, 19304354, 22519146, 27199245, 20890311, 22977256, 21142837
         20869721, 24555417, 22179537, 21756699, 20217801, 18819908, 22760595
         25483815, 23007241, 19593445, 21080143, 27351628, 20031873, 18618122
         24737581, 26784509, 24739928, 18966843, 19077215, 20704450, 19068970
         20543011, 19023822, 24713381, 20432873, 21756677, 20328248, 18674047
         18849537, 25459958, 20315311, 22897344, 27534509, 25178179, 19308965
         18948177, 19468991, 20868862, 21780146, 20466628, 21756661, 20397490
         23315153, 19706965, 20302006, 24831514, 23240358, 22178855, 19032777
         20862087, 19329654, 18974476, 20603378, 20859910, 19307662, 21847223
         20281121, 19075256, 19076343, 18866977, 22808310, 25635149, 20844426
         20904530, 20441797, 21442094, 25079710, 24674955, 18840932, 18740837
         20294666, 25602488, 21517440, 22062517, 27337759, 19174942, 20671094
         21889720, 18411216, 20117253, 24386767, 20641666, 25264559, 22092979
         21625179, 20879709, 23003979, 20165574, 19272708, 19547370, 22624709
         23084507, 20228093, 21281532, 19805359, 19461270, 19434529, 18799063
         20378086, 17008068, 21246723, 20831538, 20424899, 20361671, 18674024
         19689979, 24411921, 19873610, 16619249, 20562898, 21641414, 21091431
         19440586, 22757364, 22175564, 21241052, 20725343, 19561643, 20736227
         19399918, 19195895, 20830459, 20017509, 25790353, 21828126, 21665897
         25555252, 20746251, 25764020, 25612095, 25357142, 23096938, 19067244
         18043064, 21329301, 18885870, 26243698, 26187943, 20324049, 19536415
         23709062, 28174827, 20446883, 27314206, 21299490, 25313154, 21744290
         18254023, 20591183, 27847259, 19185876, 27207110, 22465352, 24326444
         20402832, 19627012, 27441326, 27620950, 16863642, 19639483, 19315691
         21479753, 19174521, 20401975, 18306996, 18851894, 27034890, 20581111
         20318889, 20936731, 21060755, 26828994, 22256560, 19188927, 27229389
         24570598, 25475853, 21172913, 17655240, 21266085, 19028800, 19035573
         19366375, 24523374, 25034396, 19289642, 21291274, 18007682, 23521523
         20475845, 22148226, 22528741, 25417958, 24652769, 26088426, 19326908
         19597583, 17414008, 23019710, 20897759, 26822620, 22046677, 20938170
         24825843, 19891090, 21960504, 26318627, 24509056, 19054077, 26262953
         22657942, 20428621, 21899588, 19723336, 19835133, 17532734, 19333670
         21842017, 19285025, 21373473, 23260854, 19687159, 23061453, 14643995
         20977794, 20734332, 17551063, 27548131, 21977392, 24461826, 19676012
         20588502, 23315889, 19520602, 23053606, 19841800, 20245930, 19001359
         21476308, 26546754, 19393542, 23533524, 21099555, 25429959, 19141838
         19644859, 21915719, 19908836, 21421886, 19358317, 19524158, 23548817
         25861398, 20803014, 23025340, 19335438, 19058490, 19207117, 23642282
         18799993, 25919622, 26569225, 20835241, 24662775, 19475971, 18967382
         20347562, 20348653, 19896336, 24812585, 20048359, 21896069, 19524384
         25392535, 21147908, 20440930, 25789277, 19171086, 24718260, 17867700
         19791273, 21241829, 19591608, 22707244, 18419520, 22296366, 18914624
         19571367, 22654475, 21522582, 19501299, 20425790, 19708342, 27997875
         16870214, 18202441, 24415926, 18743542, 19001390, 21875360, 25091141
         28000269, 19149990, 20382309, 22855193, 16777441, 19606174, 20848335
         25495682, 19382851, 20528052, 22762046, 24563422, 23125826, 22503297
         25192729, 23338911, 22730454, 19176326, 19048007, 18849970, 21532755
         20860659, 22905130, 21263635, 22160989, 18499088, 21059919, 18952989
         22894949, 22518784, 25856821, 25484507, 20794034, 19468347, 17533661
         19883092, 20657441, 24401351, 21285458, 18051556, 25330273, 19699191
         24437510, 20669434, 18964978, 22972770, 20828947, 21373076, 25551676
         25492379, 14283239, 25766822, 22922076, 25575628, 20368850, 21239530
         20437153, 24848928, 20880215, 20798891, 25606091, 19013183, 21133343
         22695831, 24365589, 25634317, 19587324, 20273319, 18542562, 26758193
         21063322, 22062026, 20134339, 22077517, 22815955, 24690216, 22507210
         20101006, 16354467, 21795111, 27938623, 23501901, 18797519, 25879984
         21260397, 25029423, 19354335, 19730508, 22366558, 26658759, 6599380
         20717359, 24321547, 21297872, 18964939, 26366517, 21913183, 22366322
         20171986, 20603431, 21132297, 25957038, 21542577, 22507234, 23170620
         24719736, 25600342, 18868646, 20627866, 26637824, 18110491, 16923858
         24642295, 19518079, 20466322, 25823754, 25110233, 24908321, 20842388
         17274537, 26575788, 20474192, 21644640, 21794615, 18899974, 20471920
         22806698, 19052488, 19503821, 24350620, 20074391, 19157754, 21220620
         24316947, 19865345, 19065556, 22816287, 25947799, 20878790, 23492665
         21322887, 22305887, 20879889, 24350831, 19578350, 19363645, 21072646
         20898391, 19291380, 27060167, 27086138, 22536802, 22087683, 21656630
         20373598, 19248799, 22707866, 19155797, 19279273, 18886413, 25490238
         20922010, 19990037, 25150925, 20509482, 24717859, 20703000, 22862134
         21526048, 24929210, 24560906, 20144308, 21620471, 19670108, 19068610
         20267166, 25123585, 20476175, 18549238, 22950945, 19385656, 23528412
         19684504, 21174504, 20899461, 20557786, 21911701, 19143550, 20118035
         19024808, 25760195, 20009833, 19604659, 16359751, 26039623, 19928926
         23314180, 20212067, 24737403, 20480209, 26430737, 20856766, 27169796
         21668627, 20877664, 19487147, 23149541, 24577566, 19430401, 19676905
         20925795, 21296029, 21629064, 23229229, 22865673, 20708701, 25353983
         19280225, 21315084, 19213447, 19989009, 18191823, 27314390, 25775213
         24393981, 25639019, 17319928, 19703301, 21626377, 20122715, 6418158
         23105538, 26198926, 19258504, 21188532, 23151677, 17890099, 21649497
         26446098, 16887946, 26024732, 18791688, 19721304, 19490948, 27012701
         19619732, 21164318, 18090142, 21641760, 19818513, 20139391, 24693382
         19978542, 23543183, 22165897, 22359063, 19409212, 23035249, 18990693
         20470877, 21422580, 21632821, 22351572, 20235511, 23220453, 18604493
         23008056, 22901797, 18610915, 20832516, 24801152, 26089440, 20907061
         20505778, 19183343, 21787056, 21273804, 25093739, 17835294, 24413809
         18371441, 26714910, 24385983, 20413820, 24421668, 25897615, 25643931
         23195445, 21281607, 20513399, 20558005, 20093776, 18909599, 20618595
         23572982, 19211433, 20331945, 19512341, 22256431, 19637186, 19022470
         18607546, 24573817, 19649152, 23115139, 19201867, 21294938, 20898997
         18510194, 21842740, 22454326, 24683149, 19534363, 25489607
    
    
    
    --------------------------------------------------------------------------------
    
    OPatch succeeded.
    [oracle@srv4 OPatch]$
    
    • STEP 6
    Start Standby database in  mount state 
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area 1795162112 bytes
    Fixed Size                  2925456 bytes
    Variable Size             620760176 bytes
    Database Buffers         1157627904 bytes
    Redo Buffers               13848576 bytes
    Database mounted.
    SQL> exit
    Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
    [oracle@srv4 ~]$ lsnrctl start
    
    LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 06-JUL-2021 12:43:13
    
    Copyright (c) 1991, 2014, Oracle.  All rights reserved.
    
    Starting /u01/app/oracle/product/12.1.0/dbhome_1/bin/tnslsnr: please wait...
    
    TNSLSNR for Linux: Version 12.1.0.2.0 - Production
    System parameter file is /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
    Log messages written to /u01/app/oracle/diag/tnslsnr/srv4/listener/alert/log.xml
    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=srv4.example.com)(PORT=1521)))
    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
    
    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=srv4.example.com)(PORT=1521)))
    STATUS of the LISTENER
    ------------------------
    Alias                     LISTENER
    Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
    Start Date                06-JUL-2021 12:43:15
    Uptime                    0 days 0 hr. 0 min. 0 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
    Listener Log File         /u01/app/oracle/diag/tnslsnr/srv4/listener/alert/log.xml
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=srv4.example.com)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
    The listener supports no services
    The command completed successfully
    [oracle@srv4 ~]$
    
    • STEP 7
    Apply patch in primary database 

    Shutdown primary database take tar level backup of oracle home  as mentioned above in STEP 1-2

    Check opatch lsinventory details below
     
    Check opatch lsinventory 
    
    [oracle@srv3 OPatch]$ ./opatch lsinventory
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_12-47-59PM_1.log
    
    Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2021-07-06_12-47-59PM.txt
    --------------------------------------------------------------------------------
    Local Machine Information::
    Hostname: srv3.example.com
    ARU platform id: 226
    ARU platform description:: Linux x86-64
    
    Installed Top-level Products (1):
    
    Oracle Database 12c                                                  12.1.0.2.0
    There are 1 products installed in this Oracle Home.
    
    
    There are no Interim patches installed in this Oracle Home.
    
    
    --------------------------------------------------------------------------------
    
    OPatch succeeded.
    [oracle@srv3 OPatch]$
    
    • Stop primary database  and listener
     
    
    [oracle@srv3 ~]$ lsnrctl stop
    
    LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 06-JUL-2021 12:44:27
    
    Copyright (c) 1991, 2014, Oracle.  All rights reserved.
    
    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=srv3.example.com)(PORT=1521)))
    The command completed successfully
    [oracle@srv3 ~]$ sqlplus / as sysdba
    
    SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 6 12:44:40 2021
    
    Copyright (c) 1982, 2014, Oracle.  All rights reserved.
    
    SQL> select name,open_mode,database_role from v$database;
    
    NAME      OPEN_MODE            DATABASE_ROLE
    --------- -------------------- ----------------
    ZABBIX    READ WRITE           PRIMARY
    
    SQL> shut immediate;
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL>
    
    
    • Take tar level backup of oracle home  as mentioned above in STEP 1
    • STEP 8
    SET ORACLE HOME 
    export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
    export PATH=$ORACLE_HOME/OPatch:$PATH:$ORACLE_HOME/bin
    export PATH=$ORACLE_HOME/perl/bin:$PATH
    • STEP 9 
    Set oracle home path and after unziping patch check DB patch conflict status 

    Determine whether any currently installed interim patches conflict with the patch being installed, 28259833
     
    
    
    [oracle@srv3 28259833]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    PREREQ session
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_12-52-16PM_1.log
    
    Invoking prereq "checkconflictagainstohwithdetail"
    
    Prereq "checkConflictAgainstOHWithDetail" passed.
    
    OPatch succeeded.
    
    • STEP 10
    Set your current directory to the directory where the patch is located and then run the OPatch utility by entering the following commands: 

    opatch apply  -analyze  (It will skip orginal patch apply In case error it will report)

    opatch apply  
     
    [oracle@srv3 28259833]$ opatch apply -analyze
    
    
    [oracle@srv3 28259833]$ opatch apply
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_12-55-48PM_1.log
    
    Verifying environment and performing prerequisite checks...
    OPatch continues with these patches:   19769480  20299023  20831110  21359755  21948354  22291127  23054246  24006101  24732082  25171037  25755742  26609783  26713565  26925311  27338041  27547329  28259833
    
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    All checks passed.
    
    Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
    (Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')
    
    
    Is the local system ready for patching? [y|n]
    y
    User Responded with: Y
    Backing up files...
    Applying sub-patch '19769480' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
    
    Patching component oracle.xdk, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms.util, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.xdk.parser.java, 12.1.0.2.0...
    
    Patching component oracle.oraolap, 12.1.0.2.0...
    
    Patching component oracle.xdk.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
    Applying sub-patch '20299023' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf.ic, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
    Applying sub-patch '20831110' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.oraolap.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    Applying sub-patch '21359755' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.assistants.server, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    Applying sub-patch '21948354' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
    
    Patching component oracle.xdk, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.xdk.parser.java, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.xdk.rsf, 12.1.0.2.0...
    Applying sub-patch '22291127' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.ldap.client, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.oraolap, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.oraolap.dbscripts, 12.1.0.2.0...
    Applying sub-patch '23054246' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.rdbms.dv, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.install.deinstalltool, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
    
    Patching component oracle.oracore.rsf, 12.1.0.2.0...
    
    Patching component oracle.ctx, 12.1.0.2.0...
    
    Patching component oracle.xdk, 12.1.0.2.0...
    
    Patching component oracle.nlsrtl.rsf, 12.1.0.2.0...
    
    Patching component oracle.xdk.parser.java, 12.1.0.2.0...
    
    Patching component oracle.ctx.atg, 12.1.0.2.0...
    Applying sub-patch '24006101' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.sqlplus, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.network.listener, 12.1.0.2.0...
    
    Patching component oracle.network.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dv, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.sqlplus.ic, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    Applying sub-patch '24732082' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.rdbms.install.plugins, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    Applying sub-patch '25171037' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.sqlplus.ic, 12.1.0.2.0...
    
    Patching component oracle.sqlplus, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.ordim.client, 12.1.0.2.0...
    
    Patching component oracle.ordim.client, 12.1.0.2.0...
    
    Patching component oracle.ordim.jai, 12.1.0.2.0...
    
    Patching component oracle.ordim.server, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms.install.plugins, 12.1.0.2.0...
    
    Patching component oracle.javavm.containers, 12.1.0.2.0...
    Applying sub-patch '25755742' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ] , [ oracle.has.crs, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.network.rsf, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.ldap.client, 12.1.0.2.0...
    
    Patching component oracle.oracore.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.network.listener, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    Applying sub-patch '26609783' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.oracore.rsf, 12.1.0.2.0...
    Applying sub-patch '26713565' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ] , [ oracle.has.crs, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.ldap.client, 12.1.0.2.0...
    
    Patching component oracle.rdbms.crs, 12.1.0.2.0...
    
    Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
    
    Patching component oracle.xdk, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.xdk.parser.java, 12.1.0.2.0...
    
    Patching component oracle.xdk.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    Applying sub-patch '26925311' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.network.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    Applying sub-patch '27338041' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.assistants.asm, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.network.rsf, 12.1.0.2.0...
    
    Patching component oracle.assistants.server, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.sqlplus.rsf, 12.1.0.2.0...
    Applying sub-patch '27547329' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    
    Patching component oracle.network.rsf, 12.1.0.2.0...
    
    Patching component oracle.tfa, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.ldap.rsf, 12.1.0.2.0...
    
    Patching component oracle.ctx.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.oracore.rsf, 12.1.0.2.0...
    
    Patching component oracle.ctx, 12.1.0.2.0...
    
    Patching component oracle.nlsrtl.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    Applying sub-patch '28259833' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.oracore.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rsf, 12.1.0.2.0...
    
    Patching component oracle.rdbms.rman, 12.1.0.2.0...
    
    Patching component oracle.rdbms.crs, 12.1.0.2.0...
    
    OPatch found the word "error" in the stderr of the make command.
    Please look at this stderr. You can re-run this make command.
    Stderr output:
    chmod: changing permissions of `/u01/app/oracle/product/12.1.0/dbhome_1/bin/extjobO': Operation not permitted
    make: [iextjob] Error 1 (ignored)
    
    
    Composite patch 28259833 successfully applied.
    OPatch Session completed with warnings.
    Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_12-55-48PM_1.log
    
    OPatch completed with warnings.
    [oracle@srv3 28259833]$
    
    • STEP 11
          Startup  Primary database and listener 
           
           Run the Datapatch Utility datapatch -verbose 
    [oracle@srv3 28259833]$ sqlplus /nolog
    
    SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 6 13:14:10 2021
    
    Copyright (c) 1982, 2014, Oracle.  All rights reserved.
    
    SQL> Connect / as sysdba
    Connected to an idle instance.
    SQL> startup
    ORACLE instance started.
    
    Total System Global Area 1795162112 bytes
    Fixed Size                  2925456 bytes
    Variable Size             620760176 bytes
    Database Buffers         1157627904 bytes
    Redo Buffers               13848576 bytes
    Database mounted.
    Database opened.
    SQL> alter pluggable database all open; ( In Case of Single/Multitenant (CDB/PDB))
    SQL> quit
    Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
    
    
    [oracle@srv3 28259833]$ cd $ORACLE_HOME/OPatch
    [oracle@srv3 OPatch]$ ./datapatch -verbose
    [oracle@srv3 28259833]$ cd $ORACLE_HOME/OPatch
    [oracle@srv3 OPatch]$ ./datapatch -verbose
    SQL Patching tool version 12.1.0.2.0 Production on Tue Jul  6 13:16:37 2021
    Copyright (c) 2012, 2016, Oracle.  All rights reserved.
    
    Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_9248_2021_07_06_13_16_37/sqlpatch_invocation.log
    
    Connecting to database...OK
    Note:  Datapatch will only apply or rollback SQL fixes for PDBs
           that are in an open state, no patches will be applied to closed PDBs.
           Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
           (Doc ID 1585822.1)
    Bootstrapping registry and package to current versions...done
    Determining current state...done
    
    Current state of SQL patches:
    Bundle series PSU:
      ID 181016 in the binary registry and not installed in any PDB
    
    Adding patches to installation queue and performing prereq checks...
    Installation queue:
      For the following PDBs: CDB$ROOT PDB$SEED
        Nothing to roll back
        The following patches will be applied:
          28259833 (DATABASE PATCH SET UPDATE 12.1.0.2.181016)
    
    Installing patches...
    Patch installation complete.  Total patches installed: 2
    
    Validating logfiles...
    Patch 28259833 apply (pdb CDB$ROOT): SUCCESS
      logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/28259833/22488632/28259833_apply_ZABBIX_CDBROOT_2021Jul06_13_18_17.log (no errors)
    Patch 28259833 apply (pdb PDB$SEED): SUCCESS
      logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/28259833/22488632/28259833_apply_ZABBIX_PDBSEED_2021Jul06_13_21_34.log (no errors)
    SQL Patching tool complete on Tue Jul  6 13:26:07 2021
    [oracle@srv3 OPatch]$ ./datapatch -verbose
    SQL Patching tool version 12.1.0.2.0 Production on Tue Jul  6 13:26:31 2021
    Copyright (c) 2012, 2016, Oracle.  All rights reserved.
    
    Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_10204_2021_07_06_13_26_31/sqlpatch_invocation.log
    
    Connecting to database...OK
    Note:  Datapatch will only apply or rollback SQL fixes for PDBs
           that are in an open state, no patches will be applied to closed PDBs.
           Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
           (Doc ID 1585822.1)
    Bootstrapping registry and package to current versions...done
    Determining current state...done
    
    Current state of SQL patches:
    Bundle series PSU:
      ID 181016 in the binary registry and ID 181016 in PDB CDB$ROOT, ID 181016 in PDB PDB$SEED
    
    Adding patches to installation queue and performing prereq checks...
    Installation queue:
      For the following PDBs: CDB$ROOT PDB$SEED
        Nothing to roll back
        Nothing to apply
      For the following PDBs: PDB1
        Nothing to roll back
        The following patches will be applied:
          28259833 (DATABASE PATCH SET UPDATE 12.1.0.2.181016)
    
    Installing patches...
    
    Patch installation complete.  Total patches installed: 1
    
    Validating logfiles...
    Patch 28259833 apply (pdb PDB1): SUCCESS
      logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/28259833/22488632/28259833_apply_ZABBIX_PDB1_2021Jul06_13_28_13.log (no errors)
    SQL Patching tool complete on Tue Jul  6 13:30:42 2021
    [oracle@srv3 OPatch]$
    [oracle@srv3 OPatch]$
    
    • STEP 12

      CHECK STANDBY DATABASE and apply MRP 
     
    [oracle@srv4 ~]$ sqlplus / as sysdba
    
    SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 6 13:41:20 2021
    
    Copyright (c) 1982, 2014, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
    
    SQL> alter database recover managed standby database disconnect from session;
    
    Database altered.
    
    SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH, (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL WHERE ARCH.THREAD# = APPL.THREAD#;
    
        Thread Last Sequence Received Last Sequence Applied Difference
    ---------- ---------------------- --------------------- ----------
             1                     31                    29          2
    
    
    SQL> /
    
        Thread Last Sequence Received Last Sequence Applied Difference
    ---------- ---------------------- --------------------- ----------
             1                     31                    29          2
    
    SQL> /
    
        Thread Last Sequence Received Last Sequence Applied Difference
    ---------- ---------------------- --------------------- ----------
             1                     31                    31          0
    
    SQL>
    
    
    • Patch 28440711 - Oracle JavaVM Component 12.1.0.2.181016 Database PSU
    • STEP 13
    Apply first on standby database then primary database  

    Set oracle home and check pre conflict requestories check same as in STEP  1-2  

    opatch apply 
     
    [oracle@srv4 28440711]$ opatch apply
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_14-59-47PM_1.log
    
    Verifying environment and performing prerequisite checks...
    OPatch continues with these patches:   28440711
    
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    All checks passed.
    
    Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
    (Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')
    
    
    Is the local system ready for patching? [y|n]
    y
    User Responded with: Y
    Backing up files...
    Applying interim patch '28440711' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.sqlj, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.javavm.server, 12.1.0.2.0...
    
    Patching component oracle.javavm.server.core, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.javavm.client, 12.1.0.2.0...
    
    Patching component oracle.dbjava.jdbc, 12.1.0.2.0...
    
    Patching component oracle.dbjava.ic, 12.1.0.2.0...
    Patch 28440711 successfully applied.
    Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_14-59-47PM_1.log
    
    OPatch succeeded.
    [oracle@srv4 28440711]$
    
    
    • STEP 14 
    Startup listener and standby database in mount state 
     
    
    [oracle@srv4 28440711]$ sqlplus / as sysdba
    
    SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 6 16:34:22 2021
    
    Copyright (c) 1982, 2014, Oracle.  All rights reserved.
    
    Connected to an idle instance.
    
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area 1795162112 bytes
    Fixed Size                  2925456 bytes
    Variable Size             620760176 bytes
    Database Buffers         1157627904 bytes
    Redo Buffers               13848576 bytes
    Database mounted.
    SQL> exit
    Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
    [oracle@srv4 28440711]$ lsnrctl start
    
    LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 06-JUL-2021 16:34:43
    
    Copyright (c) 1991, 2014, Oracle.  All rights reserved.
    
    Starting /u01/app/oracle/product/12.1.0/dbhome_1/bin/tnslsnr: please wait...
    
    TNSLSNR for Linux: Version 12.1.0.2.0 - Production
    System parameter file is /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
    Log messages written to /u01/app/oracle/diag/tnslsnr/srv4/listener/alert/log.xml
    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=srv4.example.com)(PORT=1521)))
    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
    
    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=srv4.example.com)(PORT=1521)))
    STATUS of the LISTENER
    ------------------------
    Alias                     LISTENER
    Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
    Start Date                06-JUL-2021 16:34:44
    Uptime                    0 days 0 hr. 0 min. 0 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
    Listener Log File         /u01/app/oracle/diag/tnslsnr/srv4/listener/alert/log.xml
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=srv4.example.com)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
    The listener supports no services
    The command completed successfully
    [oracle@srv4 28440711]$
    
    
    • STEP 15 
    Set oracle home and check pre conflict requestories check same as in STEP 2  

     
    [oracle@srv3 28440711]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    PREREQ session
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_16-45-19PM_1.log
    
    Invoking prereq "checkconflictagainstohwithdetail"
    
    Prereq "checkConflictAgainstOHWithDetail" passed.
    
    OPatch succeeded.
    [oracle@srv3 28440711]$ opatch apply -analyze
    
    
    • STEP 16
    APLLY PATCH OJVM to primary database
     
    
    [oracle@srv3 28440711]$ opatch apply
    Oracle Interim Patch Installer version 12.2.0.1.21
    Copyright (c) 2021, Oracle Corporation.  All rights reserved.
    
    
    Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
    Central Inventory : /u01/app/oraInventory
       from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
    OPatch version    : 12.2.0.1.21
    OUI version       : 12.1.0.2.0
    Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_16-47-31PM_1.log
    
    Verifying environment and performing prerequisite checks...
    OPatch continues with these patches:   28440711
    
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    All checks passed.
    
    Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
    (Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')
    
    
    Is the local system ready for patching? [y|n]
    y
    User Responded with: Y
    Backing up files...
    Applying interim patch '28440711' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'
    ApplySession: Optional component(s) [ oracle.sqlj, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.
    
    Patching component oracle.javavm.server, 12.1.0.2.0...
    
    Patching component oracle.javavm.server.core, 12.1.0.2.0...
    
    Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
    
    Patching component oracle.rdbms, 12.1.0.2.0...
    
    Patching component oracle.javavm.client, 12.1.0.2.0...
    
    Patching component oracle.dbjava.jdbc, 12.1.0.2.0...
    
    Patching component oracle.dbjava.ic, 12.1.0.2.0...
    Patch 28440711 successfully applied.
    Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-06_16-47-31PM_1.log
    
    OPatch succeeded.
    [oracle@srv3 28440711]$
    
    
    • STEP 17 
    Start Primary  database 
     
     

    [oracle@srv3 OPatch]$ sqlplus /nolog


    SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 6 17:06:14 2021


    Copyright (c) 1982, 2014, Oracle.  All rights reserved.


    SQL> Connect / as sysdba

    Connected.

    SQL> shutdown

    Database closed.

    Database dismounted.

    ORACLE instance shut down.

    SQL> startup

    ORACLE instance started.


    Total System Global Area 1795162112 bytes

    Fixed Size                  2925456 bytes

    Variable Size             704646256 bytes

    Database Buffers         1073741824 bytes

    Redo Buffers               13848576 bytes

    Database mounted.

    Database opened.

    SQL> alter pluggable database all open;


    Pluggable database altered.


    SQL>

    • STEP 18 
    lsninventory  
     
     

    [oracle@srv3 ~]$ cd /u01/app/oracle/product/12.1.0/dbhome_1/OPatch

    [oracle@srv3 OPatch]$ ./opatch lsinventory Oracle Interim Patch Installer version 12.2.0.1.21 Copyright (c) 2021, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.21 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2021-07-08_20-16-59PM_1.log Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2021-07-08_20-16-59PM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: srv3.example.com ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Database 12c 12.1.0.2.0 There are 1 products installed in this Oracle Home. Interim patches (2) : Patch 28440711 : applied on Tue Jul 06 16:50:20 IST 2021 Unique Patch ID: 22440644 Patch description: "Database PSU 12.1.0.2.181016, Oracle JavaVM Component (OCT2018)" Created on 12 Sep 2018, 13:15:28 hrs PST8PDT Bugs fixed: 28440711, 28502128, 28502113, 27923320, 27952584, 27952577, 27642235 27475603, 27461789, 27461842, 25649873, 27001733, 27000663, 27000690 26635845, 26637592, 26570171, 26027162, 26023002, 26023025, 25437695 25494413, 25494379, 24917972, 25067795, 24534298, 25076732, 25076756 24315824, 21659726, 24448240, 24448282, 23177536, 22675136, 23265914 23265965, 23727148, 22674709, 22670413, 22670385, 21188537, 22139226 22118835, 22118851, 21555660, 21811517, 19623450, 21566993, 21566944 19176885, 21068507, 21047803, 21047766, 20415564, 20408829, 20408866 19877336, 19855285, 19909862, 19895362, 19895326, 19153980, 19231857 19223010, 19245191, 19699946 Patch 28259833 : applied on Tue Jul 06 13:02:36 IST 2021 Unique Patch ID: 22488632 Patch description: "Database Patch Set Update : 12.1.0.2.181016 (28259833)" Created on 8 Oct 2018, 06:11:15 hrs PST8PDT Sub-patch 27547329; "Database Patch Set Update : 12.1.0.2.180717 (27547329)" Sub-patch 27338041; "Database Patch Set Update : 12.1.0.2.180417 (27338041)" Sub-patch 26925311; "Database Patch Set Update : 12.1.0.2.180116 (26925311)" Sub-patch 26713565; "Database Patch Set Update : 12.1.0.2.171017 (26713565)" Sub-patch 26609783; "Database Patch Set Update : 12.1.0.2.170814 (26609783)" Sub-patch 25755742; "Database Patch Set Update : 12.1.0.2.170718 (25755742)" Sub-patch 25171037; "Database Patch Set Update : 12.1.0.2.170418 (25171037)" Sub-patch 24732082; "Database Patch Set Update : 12.1.0.2.170117 (24732082)" Sub-patch 24006101; "Database Patch Set Update : 12.1.0.2.161018 (24006101)" Sub-patch 23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)" Sub-patch 22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)" Sub-patch 21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)" Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)" Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)" Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)" Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)" Bugs fixed: 19309466, 19902195, 18250893, 25437699, 19383839, 16756406, 18456643 26546664, 22364044, 18845653, 19915271, 20172151, 18417036, 23713236 24796092, 23140259, 19243521, 19658708, 18272672, 21153266, 19174430 22243719, 20688221, 20493163, 21387964, 13542050, 22734547, 21623164 19012119, 19932634, 19869255, 22232606, 18681056, 23324000, 25427662 22068305, 24589081, 19439759, 19303936, 22916353, 24835538, 22353346 21106027, 26444887, 23088803, 22529728, 26256131, 19134173, 20447445 21188584, 19390567, 26513709, 25780343, 19769480, 21097043, 21225209 26245237, 20677396, 19284031, 19450314, 19016730, 20919320, 22075064 22551446, 22721409, 18440095, 22496904, 16439813, 18354830, 20596234 22022760, 20936905, 23197103, 21514877, 26111842, 18990023, 22492533 20173897, 24624166, 17210525, 21260431, 20181030, 25056052, 19370504 21868720, 23068169, 19124589, 19402853, 19888853, 24341675, 17722075 20882568, 25653109, 23026585, 18604692, 20717081, 25546608, 27370965 19081128, 22173980, 23514710, 19178851, 20951038, 22168163, 25161298 20569094, 24308635, 19791377, 19050649, 20920911, 19189525, 19469538 27052607, 20598042, 22458049, 18988834, 23302839, 25307368, 17409174 22729345, 22842151, 19238590, 16941434, 20387265, 24397438, 20673810 23108128, 20356733, 22380919, 18436647, 23065323, 20825533, 19124336 22294260, 24790914, 20284155, 25539063, 17365043, 25914276, 20952966 22961508, 19176223, 21300341, 23237313, 18288842, 22353199, 22083366 25670786, 21419850, 26898563, 19577410, 23294548, 24737064, 19931709 25423453, 25547060, 23533807, 27726780, 24600330, 25600421, 18122373 20043616, 23124895, 18856999, 21450666, 18893947, 20076781, 26633558 26029780, 21196809, 21354456, 23725036, 20464614, 19562381, 24808595 27375542, 19189317, 25669791, 18307021, 21917884, 19708632, 27213224 25633101, 20711718, 18973548, 25982666, 19718981, 22826718, 25655390 23567857, 21773465, 20250147, 19197175, 26263721, 19597439, 21387128 22007324, 19180770, 19879746, 21785691, 20424183, 24285405, 26544823 20322560, 22228324, 23172924, 22520320, 21575362, 25058080, 22365117 22645009, 25165496, 18774543, 20124446, 21429602, 26153977, 19371175 21863727, 18940497, 19074147, 22923409, 25489342, 21380789, 19154375 19044962, 19532017, 19662635, 22374754, 20560611, 25654936, 21492036 18705806, 19578247, 22024071, 22238921, 22809871, 21184223, 23089357 19404068, 18921743, 19065677, 19018447, 19018206, 18308268, 19777862 22223463, 19304354, 22519146, 27199245, 20890311, 22977256, 21142837 20869721, 24555417, 22179537, 21756699, 20217801, 18819908, 22760595 25483815, 23007241, 19593445, 21080143, 27351628, 20031873, 18618122 24737581, 26784509, 24739928, 18966843, 19077215, 20704450, 19068970 20543011, 19023822, 24713381, 20432873, 21756677, 20328248, 18674047 18849537, 25459958, 20315311, 22897344, 27534509, 25178179, 19308965 18948177, 19468991, 20868862, 21780146, 20466628, 21756661, 20397490 23315153, 19706965, 20302006, 24831514, 23240358, 22178855, 19032777 20862087, 19329654, 18974476, 20603378, 20859910, 19307662, 21847223 20281121, 19075256, 19076343, 18866977, 22808310, 25635149, 20844426 20904530, 20441797, 21442094, 25079710, 24674955, 18840932, 18740837 20294666, 25602488, 21517440, 22062517, 27337759, 19174942, 20671094 21889720, 18411216, 20117253, 24386767, 20641666, 25264559, 22092979 21625179, 20879709, 23003979, 20165574, 19272708, 19547370, 22624709 23084507, 20228093, 21281532, 19805359, 19461270, 19434529, 18799063 20378086, 17008068, 21246723, 20831538, 20424899, 20361671, 18674024 19689979, 24411921, 19873610, 16619249, 20562898, 21641414, 21091431 19440586, 22757364, 22175564, 21241052, 20725343, 19561643, 20736227 19399918, 19195895, 20830459, 20017509, 25790353, 21828126, 21665897 25555252, 20746251, 25764020, 25612095, 25357142, 23096938, 19067244 18043064, 21329301, 18885870, 26243698, 26187943, 20324049, 19536415 23709062, 28174827, 20446883, 27314206, 21299490, 25313154, 21744290 18254023, 20591183, 27847259, 19185876, 27207110, 22465352, 24326444 20402832, 19627012, 27441326, 27620950, 16863642, 19639483, 19315691 21479753, 19174521, 20401975, 18306996, 18851894, 27034890, 20581111 20318889, 20936731, 21060755, 26828994, 22256560, 19188927, 27229389 24570598, 25475853, 21172913, 17655240, 21266085, 19028800, 19035573 19366375, 24523374, 25034396, 19289642, 21291274, 18007682, 23521523 20475845, 22148226, 22528741, 25417958, 24652769, 26088426, 19326908 19597583, 17414008, 23019710, 20897759, 26822620, 22046677, 20938170 24825843, 19891090, 21960504, 26318627, 24509056, 19054077, 26262953 22657942, 20428621, 21899588, 19723336, 19835133, 17532734, 19333670 21842017, 19285025, 21373473, 23260854, 19687159, 23061453, 14643995 20977794, 20734332, 17551063, 27548131, 21977392, 24461826, 19676012 20588502, 23315889, 19520602, 23053606, 19841800, 20245930, 19001359 21476308, 26546754, 19393542, 23533524, 21099555, 25429959, 19141838 19644859, 21915719, 19908836, 21421886, 19358317, 19524158, 23548817 25861398, 20803014, 23025340, 19335438, 19058490, 19207117, 23642282 18799993, 25919622, 26569225, 20835241, 24662775, 19475971, 18967382 20347562, 20348653, 19896336, 24812585, 20048359, 21896069, 19524384 25392535, 21147908, 20440930, 25789277, 19171086, 24718260, 17867700 19791273, 21241829, 19591608, 22707244, 18419520, 22296366, 18914624 19571367, 22654475, 21522582, 19501299, 20425790, 19708342, 27997875 16870214, 18202441, 24415926, 18743542, 19001390, 21875360, 25091141 28000269, 19149990, 20382309, 22855193, 16777441, 19606174, 20848335 25495682, 19382851, 20528052, 22762046, 24563422, 23125826, 22503297 25192729, 23338911, 22730454, 19176326, 19048007, 18849970, 21532755 20860659, 22905130, 21263635, 22160989, 18499088, 21059919, 18952989 22894949, 22518784, 25856821, 25484507, 20794034, 19468347, 17533661 19883092, 20657441, 24401351, 21285458, 18051556, 25330273, 19699191 24437510, 20669434, 18964978, 22972770, 20828947, 21373076, 25551676 25492379, 14283239, 25766822, 22922076, 25575628, 20368850, 21239530 20437153, 24848928, 20880215, 20798891, 25606091, 19013183, 21133343 22695831, 24365589, 25634317, 19587324, 20273319, 18542562, 26758193 21063322, 22062026, 20134339, 22077517, 22815955, 24690216, 22507210 20101006, 16354467, 21795111, 27938623, 23501901, 18797519, 25879984 21260397, 25029423, 19354335, 19730508, 22366558, 26658759, 6599380 20717359, 24321547, 21297872, 18964939, 26366517, 21913183, 22366322 20171986, 20603431, 21132297, 25957038, 21542577, 22507234, 23170620 24719736, 25600342, 18868646, 20627866, 26637824, 18110491, 16923858 24642295, 19518079, 20466322, 25823754, 25110233, 24908321, 20842388 17274537, 26575788, 20474192, 21644640, 21794615, 18899974, 20471920 22806698, 19052488, 19503821, 24350620, 20074391, 19157754, 21220620 24316947, 19865345, 19065556, 22816287, 25947799, 20878790, 23492665 21322887, 22305887, 20879889, 24350831, 19578350, 19363645, 21072646 20898391, 19291380, 27060167, 27086138, 22536802, 22087683, 21656630 20373598, 19248799, 22707866, 19155797, 19279273, 18886413, 25490238 20922010, 19990037, 25150925, 20509482, 24717859, 20703000, 22862134 21526048, 24929210, 24560906, 20144308, 21620471, 19670108, 19068610 20267166, 25123585, 20476175, 18549238, 22950945, 19385656, 23528412 19684504, 21174504, 20899461, 20557786, 21911701, 19143550, 20118035 19024808, 25760195, 20009833, 19604659, 16359751, 26039623, 19928926 23314180, 20212067, 24737403, 20480209, 26430737, 20856766, 27169796 21668627, 20877664, 19487147, 23149541, 24577566, 19430401, 19676905 20925795, 21296029, 21629064, 23229229, 22865673, 20708701, 25353983 19280225, 21315084, 19213447, 19989009, 18191823, 27314390, 25775213 24393981, 25639019, 17319928, 19703301, 21626377, 20122715, 6418158 23105538, 26198926, 19258504, 21188532, 23151677, 17890099, 21649497 26446098, 16887946, 26024732, 18791688, 19721304, 19490948, 27012701 19619732, 21164318, 18090142, 21641760, 19818513, 20139391, 24693382 19978542, 23543183, 22165897, 22359063, 19409212, 23035249, 18990693 20470877, 21422580, 21632821, 22351572, 20235511, 23220453, 18604493 23008056, 22901797, 18610915, 20832516, 24801152, 26089440, 20907061 20505778, 19183343, 21787056, 21273804, 25093739, 17835294, 24413809 18371441, 26714910, 24385983, 20413820, 24421668, 25897615, 25643931 23195445, 21281607, 20513399, 20558005, 20093776, 18909599, 20618595 23572982, 19211433, 20331945, 19512341, 22256431, 19637186, 19022470 18607546, 24573817, 19649152, 23115139, 19201867, 21294938, 20898997 18510194, 21842740, 22454326, 24683149, 19534363, 25489607 -------------------------------------------------------------------------------- OPatch succeeded. [oracle@srv3 OPatch]$


    • STEP 19 
    Check database sysnc status  
     
     

    SQL> select name,open_mode,database_role from v$database;

    NAME OPEN_MODE DATABASE_ROLE --------- -------------------- ---------------- ZABBIX MOUNTED PHYSICAL STANDBY SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH, (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL WHERE ARCH.THREAD# = APPL.THREAD#; Thread Last Sequence Received Last Sequence Applied Difference ---------- ---------------------- --------------------- ---------- 1 50 48 2 SQL> alter database recover managed standby database disconnect from session; Database altered. SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH, (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL WHERE ARCH.THREAD# = APPL.THREAD#; Thread Last Sequence Received Last Sequence Applied Difference ---------- ---------------------- --------------------- ---------- 1 50 50 0 SQL> !hostname srv4.example.com SQL>