Monday 23 December 2019

[PRCF-2030 : Parallel transfer workload distribution error. 12c (12.1.0.2.0)


add node command
oracle@srv1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={srv2}"


Error:

Error while copying directory /u01/app/oracle/product/12.1.0/dbhome_2 with exclude file list '/tmp/OraInstall2019-12-23_02-59-55PM/installExcludeFile.lst' to nodes 'srv2'. [PRCF-2030 : Parallel transfer workload distribution error.
PRCF-2023 : The following contents cannot be transferred as they are non-readable.
Directories:
 
Files:
   /u01/app/oracle/product/12.1.0/dbhome_2/p6880880_121010_Linux-x86-64_17.zip]
Refer to '/u01/app/oraInventory/logs/addNodeActions2019-12-23_02-59-55PM.log' for details. You may fix the errors on the required remote nodes. Refer to the install guide for error recovery. Click 'Yes' if you want to proceed. Click 'No' to exit the install. Do you want to continue?

Solution:
Check permission on directory and sub directories.

In this case p6880880_121010_Linux-x86-64_17.zip file having root permissions and it was conflicting while copying the nodes.

Check link below:

RAC : Add Failed cluster on 2nd node 12c R1 (12.1.0.2.0)

RAC : Add Failed cluster on 2nd node 12c R1 (12.1.0.2.0)

Scenario Preview :

We have 2 node RAC configuration. 2nd node operating system got failed or crash.
Recreating n 2nd node from scratch.

DB Name :

RAC

Instance :

RAC1, RAC2

Server Name:

srv1.example.com , srv2.example.com

STEPS:
  • Removing the rdbms inventory  of 2nd node from the 1st cluster using oracle user.
  • Removing the grid inventory of 2nd node from the 1st cluster using grid user.
  • Removing the node from cluster-ware registry.
  • Check and remove VIP of node 2.
  • Remove the rac2 instance of 2nd node.
  • Remove 2nd node directory from Operating system.
  • Add node  srv2 using grid user from srv1 server
  • Deconfigured the oracle grid infrastructure on this node: srv2.
  • Run root.sh on srv2 node using root user.
  • Check VIP status on node  srv2.
  • Add node rdbms from srv1 (GUI)
  • Add and start instance of rac2 
  • Modify service , start and check services on node.
 
Summary :

STEP : 
Directory : /u01/app/oracle/product/12.1.0/dbhome_1/oui/bin

[oracle@srv1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 "CLUSTER_NODES={srv1}" LOCAL_NODE=srv1
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5981 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[oracle@srv1 bin]$ pwd
/u01/app/oracle/product/12.1.0/dbhome_1/oui/bin
[oracle@srv1 bin]$

STEP :
Directory : /u01/app/12.1.0/grid/oui/bin

[grid@srv1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.1.0/grid "CLUSTER_NODES={srv1}" CRS=TRUE -silent
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5981 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

STEP :

[root@srv1 bin]# crsctl delete node -n srv2
CRS-4661: Node srv2 successfully deleted.
[root@srv1 bin]#

STEP :

[root@srv1 bin]# srvctl stop vip -vip srv2-vip.example.com -force
[root@srv1 bin]#


[root@srv1 bin]# srvctl remove vip -vip srv2-vip.example.com
Please confirm that you intend to remove the VIPs srv2-vip.example.com (y/[n]) y
[root@srv1 bin]#

STEP :

[root@srv1 bin]# srvctl remove instance -db rac -instance rac2
Remove instance from the database rac? (y/[n]) y
[root@srv1 bin]#

#################################################################################

STEP :

[root@srv2 u01]# rm -rf app
[root@srv2 u01]# cd /tmp
[root@srv2 tmp]# ll
total 276
drwxrwx---  3 grid   oinstall   4096 Dec 20 16:26 CVU_12.1.0.2.0_grid
drwxrwx---  3 oracle oinstall   4096 Dec 23 12:40 CVU_12.1.0.2.0_oracle
drwxr-xr-x  3 grid   oinstall   4096 Dec 21 05:47 CVU_12.1.0.2.0_resource
drwxr-x---  2 grid   oinstall   4096 Dec 23 11:57 hsperfdata_grid
drwxr-xr-x  2 root   root       4096 Dec 23 12:58 hsperfdata_root
drwx------  2 root   root       4096 Oct 30 13:58 keyring-822RUe
drwx------  2 root   root       4096 Sep 27 14:46 keyring-8GhiEO
drwx------  2 root   root       4096 Oct 14 12:25 keyring-odq9s3
drwx------  2 root   root       4096 Sep 20 15:44 keyring-VqEQir
-rw-r--r--  1 root   root     222736 Dec 20 18:31 modules.dep
drwxr-xr-x  4 grid   oinstall   4096 Dec 20 17:40 OraInstall2019-12-20_04-26-29PM
drwxr-xr-x  4 oracle oinstall   4096 Dec 20 19:07 OraInstall2019-12-20_06-51-52PM
drwx------  2 gdm    gdm        4096 Dec 20 15:47 orbit-gdm
drwx------. 2 root   root       4096 Dec 19 13:23 pulse-Fhy2hgN3LpSQ
drwx------  2 gdm    gdm        4096 Dec 20 15:47 pulse-ry0r6BdeXuu1

[root@srv2 tmp]# rm -rf OraInstall2019-12-20_04-26-29PM OraInstall2019-12-20_06-51-52PM CVU_12.1.0.2.0_resource CVU_12.1.0.2.0_oracle CVU_12.1.0.2.0_grid

[root@srv2 tmp]# cd /usr/local/bin/
-rwxr-xr-x 1 grid root 6583 Dec 20 18:27 coraenv
-rwxr-xr-x 1 grid root 2445 Dec 20 18:27 dbhome
-rwxr-xr-x 1 grid root 7012 Dec 20 18:27 oraenv
[root@srv2 bin]# rm -f *

 [root@srv2 oracle]#cd /etc/oracle

[root@srv2 oracle]# ll
total 3000
drwxrwx--- 2 root oinstall    4096 Sep 23 17:27 lastgasp
drwxrwxrwt 2 root oinstall    4096 Dec 23 16:07 maps
-rw-r--r-- 1 root oinstall      72 Dec 23 14:51 ocr.loc
-rw-r--r-- 1 root root           0 Dec 23 14:51 ocr.loc.orig
-rw-r--r-- 1 root oinstall      80 Dec 23 14:51 olr.loc
-rw-r--r-- 1 root root           0 Dec 23 14:51 olr.loc.orig
drwxrwxr-x 5 root oinstall    4096 Sep 23 17:20 oprocd
drwxr-xr-x 3 root oinstall    4096 Sep 23 17:20 scls_scr
-rws--x--- 1 root oinstall 3044561 Dec 23 14:49 setasmgid

[root@srv2 oracle]#

 rm -f ocr.loc ocr.loc.orig olr.loc olr.loc.orig

#################################################################################

STEP :

[grid@srv1 ~]$ cd /u01/app/12.1.0/grid/addnode/
[grid@srv1 addnode]$ ll
total 12
-rw-r--r-- 1 grid oinstall 1963 Sep 23 16:54 addnode_oraparam.ini
-rw-r--r-- 1 grid oinstall 1971 Jul  7  2014 addnode_oraparam.ini.sbs
-rwxr-xr-x 1 grid oinstall 3575 Sep 23 16:54 addnode.sh

[grid@srv1 addnode]$ ./addnode.sh -ignoreSysPrereqs -ignorePrereq -silent "CLUSTER_NEW_NODES={srv2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={srv2-vip}"
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 20656 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 5981 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2019-12-23_01-14-13PM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.


..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/12.1.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /u01/app/12.1.0/grid/root.sh

Execute /u01/app/12.1.0/grid/root.sh on the following nodes:
[srv2]

The scripts can be executed in parallel on all the nodes.

..........
Update Inventory in progress.
..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.
[grid@srv1 addnode]$


##############################################################################

STEP :

[root@srv2 install]# pwd
/u01/app/12.1.0/grid/crs/install
[root@srv2 install]#

STEP :

[root@srv2 install]# ./rootcrs.sh -verbose -deconfig -force
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2019-12-23 14:48:33:   mkpath (/u01/app/12.1.0/grid/cfgtoollogs/crsconfig)
2019/12/23 14:48:36 CLSRSC-561: The Oracle Grid Infrastructure has already been deconfigured on this node: srv2.


###############
[root@srv2]# cd /u01/app/12.1.0/grid

STEP :

[root@srv2]# ./root.sh

[root@srv2 install]# srvctl status vip -vip srv2-vip.example.com
VIP srv2-vip is enabled
VIP srv2-vip is running on node: srv2
[root@srv2 install]#


################


/u01/app/oracle/product/12.1.0/dbhome_2/addnode
[oracle@srv1 addnode]$ ll
total 12
-rw-r--r-- 1 oracle oinstall 1963 Sep 27 15:35 addnode_oraparam.ini
-rw-r--r-- 1 oracle oinstall 1971 Jul  7  2014 addnode_oraparam.ini.sbs
-rwxr-xr-x 1 oracle oinstall 3593 Sep 27 15:35 addnode.sh
[oracle@srv1 addnode]$ 

###############

STEP :
[oracle@srv1 addnode]$ ./addnode.sh  -silent "CLUSTER_NEW_NODES={srv2}"
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 20439 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 5956 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Logfile Location : /u01/app/oraInventory/logs/sshsetup1_2019-12-23_03-32-37PM.log
ClusterLogger - log file location: /u01/app/oracle/product/12.1.0/dbhome_2/oui/bin/Logs/remoteInterfaces2019-12-23_03-32-37PM.log
Validating remote binaries..
Remote binaries check succeeded
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2019-12-23_03-32-37PM.log

####################

STEP :

[oracle@srv1 addnode]$ srvctl add instance -db rac -instance rac2 -node srv2
[oracle@srv1 addnode]$
[oracle@srv1 addnode]$
[oracle@srv1 addnode]$ srvctl start instance -db rac -instance rac2
[oracle@srv1 addnode]$ srvctl status database -d rac
Instance rac1 is running on node srv1
Instance rac2 is running on node srv2


[oracle@srv1 addnode]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Dec 23 16:05:06 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select inst_id,status from gv$instance;

   INST_ID STATUS
---------- ------------
         1 OPEN
         2 OPEN

SQL> exit


##########################################################################

STEP :

[oracle@srv1 addnode]$ srvctl modify service -db rac -service pretaf -modifyconfig -preferred "rac1,rac2"
[oracle@srv1 addnode]$ srvctl start service -db rac -service pretaf -node srv2

[oracle@srv1 addnode]$ srvctl status service -db rac
Service acsrac is running on instance(s) rac1
Service pretaf is running on instance(s) rac1,rac2



Wednesday 18 December 2019

RAC convert physical standby to snapshot database

[oracle@srv3 rmanbkp]$ . oraenv
ORACLE_SID = [stdrac] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@srv3 rmanbkp]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Fri Dec 13 17:20:58 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> ALTER SYSTEM SET db_flashback_retention_target = 2880 SCOPE = BOTH;

System altered.

SQL> sho parameter db_recovery

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /u01/app/oracle/fast_recovery_
                                                 area
db_recovery_file_dest_size           big integer 4560M
SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
RAC       MOUNTED              PHYSICAL STANDBY

SQL> select name,open_mode,database_role,db_unique_name from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    DB_UNIQUE_NAME
--------- -------------------- ---------------- ------------------------------
RAC       MOUNTED              PHYSICAL STANDBY stdrac

SQL> select process,status,sequence#,thread# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#    THREAD#
--------- ------------ ---------- ----------
ARCH      CLOSING             610          1
ARCH      CLOSING             607          1
ARCH      CONNECTED             0          0
ARCH      CLOSING             609          1
RFS       IDLE                640          2
RFS       IDLE                  0          0
RFS       IDLE                611          1
RFS       IDLE                  0          0
MRP0      WAIT_FOR_LOG        640          2

9 rows selected.

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> select process,status,sequence#,thread# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#    THREAD#
--------- ------------ ---------- ----------
ARCH      CLOSING             610          1
ARCH      CLOSING             607          1
ARCH      CONNECTED             0          0
ARCH      CLOSING             609          1
RFS       IDLE                640          2
RFS       IDLE                  0          0
RFS       IDLE                611          1
RFS       IDLE                  0          0

8 rows selected.

SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area  754974720 bytes
Fixed Size                  2928968 bytes
Variable Size             566234808 bytes
Database Buffers          180355072 bytes
Redo Buffers                5455872 bytes
Database mounted.
SQL> alter database convert to snapshot standby;

Database altered.

SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
RAC       MOUNTED              SNAPSHOT STANDBY

SQL> select name,open_mode,database_role,db_unique_name from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    DB_UNIQUE_NAME
--------- -------------------- ---------------- ------------------------------
RAC       MOUNTED              SNAPSHOT STANDBY stdrac

SQL> alter database open;

Database altered.

SQL> select name,open_mode,database_role,db_unique_name from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    DB_UNIQUE_NAME
--------- -------------------- ---------------- ------------------------------
RAC       READ WRITE           SNAPSHOT STANDBY stdrac

[oracle@srv3 rmanbkp]$ sqlplus sys/system123@stdrac as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Fri Dec 13 17:31:28 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select name,open_mode,database_role,db_unique_name from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    DB_UNIQUE_NAME
--------- -------------------- ---------------- ------------------------------
RAC       READ WRITE           SNAPSHOT STANDBY stdrac

SQL> select FILE_NAME,TABLESPACE_NAME from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
TABLESPACE_NAME
------------------------------
/u02/oradata/stdrac/system.279.1020351597
SYSTEM

/u02/oradata/stdrac/sysaux.271.1020351511
SYSAUX

/u02/oradata/stdrac/undotbs2.278.1020352203
UNDOTBS2


FILE_NAME
--------------------------------------------------------------------------------
TABLESPACE_NAME
------------------------------
/u02/oradata/stdrac/users.272.1020351691
USERS

/u02/oradata/stdrac/idata.284.1020709157
IDATA

/u02/oradata/stdrac/undotbs1.270.1020351693
UNDOTBS1


FILE_NAME
--------------------------------------------------------------------------------
TABLESPACE_NAME
------------------------------
/u02/oradata/stdrac/ilog.285.1020709245
ILOG

/u02/oradata/stdrac/indx.286.1020709561
INDX

/u02/oradata/stdrac/indx.291.1026913655
INDX


9 rows selected.

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area  754974720 bytes
Fixed Size                  2928968 bytes
Variable Size             574623416 bytes
Database Buffers          171966464 bytes
Redo Buffers                5455872 bytes
Database mounted.
SQL> alter database convert to physical standby;

Database altered.

SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
stSQL
SQL>
SQL>
SQL> startup mount;
ORACLE instance started.

Total System Global Area  754974720 bytes
Fixed Size                  2928968 bytes
Variable Size             574623416 bytes
Database Buffers          171966464 bytes
Redo Buffers                5455872 bytes
Database mounted.
SQL> select name,open_mode,database_Role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
RAC       MOUNTED              PHYSICAL STANDBY

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> select process,status,sequence#,thread# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#    THREAD#
--------- ------------ ---------- ----------
ARCH      CONNECTED             0          0
ARCH      CLOSING             623          1
ARCH      CONNECTED             0          0
ARCH      CONNECTED             0          0
RFS       IDLE                653          2
RFS       IDLE                  0          0
RFS       IDLE                624          1
RFS       IDLE                  0          0
MRP0      WAIT_FOR_LOG        653          2

9 rows selected.

SQL>  select process,status,sequence#,thread# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#    THREAD#
--------- ------------ ---------- ----------
ARCH      CONNECTED             0          0
ARCH      CLOSING             623          1
ARCH      CONNECTED             0          0
ARCH      CONNECTED             0          0
RFS       IDLE                653          2
RFS       IDLE                  0          0
RFS       IDLE                624          1
RFS       IDLE                  0          0
MRP0      WAIT_FOR_LOG        653          2

9 rows selected.

SQL>


Patch for Bug# 21255373 for platform 12.1.0.2.0 grid

Scenario Preview:

Patch for Bug# 21255373 for platform 12.1.0.2.0 grid

While applying patch we do pre-check analyze.   We found getting error as stdrac database as instance do not exists , There was service stdrac was offline and it was do not exists on clusterware services, As we remove stdrac database service error got removed.



[root@srv1 grid]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
[root@srv1 grid]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.asm
               ONLINE  ONLINE       srv1                     Started,STABLE
               ONLINE  ONLINE       srv2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv2                     169.254.106.5 192.16
                                                             8.10.2,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv2                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv2                     Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.acsrac.svc
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,STABLE
      2        ONLINE  ONLINE       srv2                     Open,STABLE
ora.rac.pretaf.svc
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.pretaf_preconnect.svc
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.stdrac.db
      1        OFFLINE OFFLINE                               STABLE
      2        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------

[root@srv1 OPatch]# ./opatchauto apply /u01/21255373/21255373 -analyze

opatchauto must run from one of the homes specified
opatchauto returns with error code = 2
[root@srv1 OPatch]# . oraenv
ORACLE_SID = [rac1] ? +ASM1
The Oracle base has been changed from /u01/app/oracle to /u01/app/grid
[root@srv1 OPatch]# ./opatchauto apply /u01/21255373/21255373 -analyze

OPatchauto session is initiated at Wed Dec 18 14:13:36 2019

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2019-12-18_02-13-41PM.log.

Following home(s) will not be included as part of current opatchauto session as it's software only RAC home:
Home path:/u01/app/12.1.0/grid, Host:srv2


Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2019-12-18_02-14-32PM.log
The id for this session is TQQV
The following RAC Databases have no instances associated with it.
        stdrac


The following RAC Databases have no instances associated with it.
        stdrac


The following RAC Databases have no instances associated with it.
        stdrac


OPATCHAUTO-72017: No RAC instance.
OPATCHAUTO-72017: RAC Databases have no instances associated with it.
OPATCHAUTO-72017: Check system configuration.
OPatchAuto failed.

OPatchauto session completed at Wed Dec 18 14:14:42 2019
Time taken to complete the session 1 minute, 6 seconds

 opatchauto failed with error code 42
[root@srv1 OPatch]#
#################################################################################
[root@srv1 grid]# srvctl remove database -d stdrac
Remove the database stdrac? (y/[n]) y
[root@srv1 grid]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.asm
               ONLINE  ONLINE       srv1                     Started,STABLE
               ONLINE  ONLINE       srv2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv2                     169.254.106.5 192.16
                                                             8.10.2,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv2                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv2                     Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.acsrac.svc
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,STABLE
      2        ONLINE  ONLINE       srv2                     Open,STABLE
ora.rac.pretaf.svc
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.pretaf_preconnect.svc
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
[root@srv1 grid]#

[root@srv1 OPatch]# ./opatchauto apply /u01/21255373/21255373 -analyze

OPatchauto session is initiated at Wed Dec 18 15:26:32 2019

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2019-12-18_03-26-37PM.log.

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2019-12-18_03-27-25PM.log
The id for this session is RJVJ

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_2
Patch applicability verified successfully on home /u01/app/12.1.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/dbhome_2


Verifying SQL patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_2
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has failed:

Host:srv1
CRS Home:/u01/app/12.1.0/grid
Version:12.1.0.2.0


==Following patches were SKIPPED:

Patch: /u01/21255373/21255373
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-12-18_15-27-57PM_1.log
Reason: /u01/21255373/21255373 is not required to be applied to oracle home /u01/app/12.1.0/grid


Host:srv1
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_2
Version:12.1.0.2.0
Analysis for patches has failed.


==Following patches FAILED in analysis for apply:

Patch: /u01/21255373/21255373
Log: /u01/app/oracle/product/12.1.0/dbhome_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-12-18_15-27-58PM_1.log
Reason: Failed during Analysis: CheckConflictAgainstOracleHome Failed, [ Prerequisite Status: FAILED, Prerequisite output:
Summary of Conflict Analysis:

There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
21255373, 29494060

Whole composite patch Conflicts/Supersets are:

Patch : 21255373

        Conflict with Composite Patch 29494060

Detail Conflicts/Supersets for each patch are:

Patch : 21255373

        Confl ...


OPATCHAUTO-72053: Analysis for the patches failed.
OPATCHAUTO-72053: Command execution failed.
OPATCHAUTO-72053: Please check the summary for more details.

OPatchauto session completed at Wed Dec 18 15:28:12 2019
Time taken to complete the session 1 minute, 41 seconds
[root@srv1 OPatch]#

NOTE: Already applied patch 29494060 so it is conflicting the same with patch number 21255373.

we can rollback 29494060 patch and apply further 21255373 patch.


Monday 16 December 2019

Creation of LOGICAL STANDBY on standalone database 12c

Creation of  LOGICAL STANDBY on standalone database 12c


Step-1 > On Standby First we have to cancel the recovery process 
SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Step-2 > Add the log_archive_dest_3 at primary database for future , when you make primary database to 
logical standby
SQL>ALTER SYSTEM SET log_archive_dest_3='LOCATION=/u01/app/oracle/oradata/archivelog/
VALID_FOR=(STANDBY_LOGFILES, STANDBY_ROLE) DB_UNIQUE_NAME=orcl';
Step-3 > Build a Dictionary in the Redo Data
SQL> EXECUTE DBMS_LOGSTDBY.BUILD;
The DBMS_LOGSTDBY.BUILD procedure uses Flashback Query to obtain a consistent snapshot
 of the data dictionary that is then logged in the redo stream. Oracle recommends setting the
UNDO_RETENTION 
initialization parameter to 3600 on both the primary and logical standby databases.

Step-4 > To continue applying redo data to the physical standby database until it is ready to convert to a 
logical standby database , issue the following SQL statement:
SQL> ALTER DATABASE RECOVER TO LOGICAL STANDBY orcldr;
Step-5> Create a new password File 
Step-6> On the logical standby database, shutdown the instance and issue the STARTUP MOUNT statement
to start and mount the database. Do not open the database; it should remain closed to user access 
 until later in the creation process
In pfile Change the ------------(db_name=orcldr)
SQL> Startup mount pfile=’?/dbs/pfile.ora’
SQL> create spfile from pfile='?/dbs/pfile.ora';
SQL> shut immediate
SQL> startup mount
Step-7 > Add the log_archive_dest_3 at Logical standby database  
SQL> ALTER SYSTEM SET log_archive_dest_3 ='LOCATION=/u01/app/oracle/oradata/archivelog/  
VALID_FOR=(STANDBY_LOGFILES, STANDBY_ROLE) DB_UNIQUE_NAME=orcldr';
Step-8 > Open the Logical Database 
SQL> ALTER DATABASE OPEN RESETLOGS;
Step-9 > Issue the following statement to begin applying redo data to the logical standby database
SQL>  ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;



Dataguard : Switchover from Pluggable primary to standby databse 12c


  • Points to be noted during switchover

1. Verify the primary database instance is open and the standby database instance is mounted
2. Verify there are no active users connected to the databases  SELECT username, status, logon_time
  FROM v$session
 WHERE status = 'ACTIVE'
3.Make sure the last redo data transmitted from the Primary database was applied on the standby database
Issue the following commands on Primary database and Standby database to find out redo's applied or not.
SQL>select sequence#, applied from v$archvied_log;


SQL>select sequence#, applied from v$archvied_log; (check on physical Standby)
In order to apply redo data to the standby database as soon as it is received, use Real-time apply
Step -1 > Command to execute on primary for Switchover to Standby 
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;
Step-2> After step 1 finishes, Switch the original physical standby db orcldb to primary role;
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;
Step-3> shut down and restart the former primary instance orcl
SQL>SHUTDOWN IMMEDIATE;
SQL>STARTUP MOUNT;

Step-4 > After step 3 completes
Note- If you are using oracle 10 g release 1 than you need to shutdown and restart the new primary 
database orcldr
SQL>SHUTDOWN IMMEDIATE;
SQL>STARTUP;
OR----- If you are using oracle 10 g release 2 than you need to open only the new primary 
database orcldr
SQL>ALTER DATABASE OPEN;


 Step-4 > To Start the Recovery of new Standby ORCL
SQL>alter database recover managed standby database disconnect from session;

Step-5 > On the new primary database orcldr, perform a SWITCH LOGFILE to start sending redo 
data to the standby database ORCL
SQL>ALTER SYSTEM SWITCH LOGFILE;

-------------------------------------------------------------------------------------------------------------

Switchover from Pluggable primary to standby database  example 


Primary Instance  : rac1,rac2

Physical standby Instance : racdr
  • In this example we have 2 node RAC with non rac DR physical standby database
  • Check archive log sync between primary and standby database script used standby.sql and d.sql during switchover  

 select sequence#,applied from v$archived_log order by sequence# asc;

SQL> !cat standby.sql

select name,open_mode,database_role from v$database;

select process,status,sequence# from v$managed_standby;

SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH, (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL WHERE ARCH.THREAD# = APPL.THREAD#;

SQL> !cat d.sql

set lines 200


select name, instance_name,db_unique_name, open_mode, database_role, flashback_on  current_scn,to_char(standby_became_primary_scn) failover_scn from v$database,v$instance;


SQL> @d


NAME   INSTANCE_NAME    DB_UNIQUE_NAME   OPEN_MODE        DATABASE_ROLE CURRENT_SCN    FAILOVER_SCN

--------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

RAC   rac1    rac   READ WRITE        PRIMARY NO    0


SQL> @standby                                                                      


NAME   OPEN_MODE        DATABASE_ROLE

--------- -------------------- ----------------

RAC   READ WRITE        PRIMARY



PROCESS   STATUS SEQUENCE#

--------- ------------ ----------

ARCH   CLOSING        57

ARCH   CLOSING        58

ARCH   CLOSING        56

ARCH   OPENING        47

DGRD   ALLOCATED 0

DGRD   ALLOCATED 0

LNS   CONNECTED 0

DGRD   ALLOCATED 0

LNS   WRITING        59

DGRD   ALLOCATED 0


10 rows selected.



    Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

 1        58      58   0

 1        58      58   0

 2        52      52   0

 2        52      52   0


  • Switchover command for converting primary database to standby database 

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;


Database altered.

  • Shutdown database and start old primary in mount stage ( In case of RAC we can start srvctl start database -d rac -o mount ) , Here when type above command both instance 1 and 2 were shutdown.) 

SQL> SHUTDOWN IMMEDIATE;

ORA-01012: not logged on

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@srv1 ~]$ sqlplus / as sysdba


SQL*Plus: Release 12.2.0.1.0 Production on Thu Dec 16 15:38:28 2021


Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to an idle instance.


SQL> startup mount;

ORACLE instance started.


Total System Global Area 2432696320 bytes

Fixed Size     8795664 bytes

Variable Size   805308912 bytes

Database Buffers  1610612736 bytes

Redo Buffers     7979008 bytes

Database mounted.

SQL> @d


NAME   INSTANCE_NAME    DB_UNIQUE_NAME   OPEN_MODE        DATABASE_ROLE CURRENT_SCN    FAILOVER_SCN

--------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

RAC   rac1    rac   MOUNTED        PHYSICAL STANDBY NO    0


  • Now apply switchover command for converting  standby database to primary database

SQL> @standby


NAME   OPEN_MODE        DATABASE_ROLE

--------- -------------------- ----------------

RAC   MOUNTED        PHYSICAL STANDBY



PROCESS   STATUS SEQUENCE#

--------- ------------ ----------

ARCH   CLOSING        47

DGRD   ALLOCATED 0

DGRD   ALLOCATED 0

ARCH   CLOSING        58

ARCH   CLOSING        52

ARCH   CLOSING        57

RFS   IDLE 0

RFS   IDLE        59

RFS   IDLE 0

RFS   IDLE 0

RFS   IDLE        53


PROCESS   STATUS SEQUENCE#

--------- ------------ ----------

RFS   IDLE 0

RFS   IDLE 0

RFS   IDLE 0

MRP0   APPLYING_LOG        53


15 rows selected.



    Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

1        58      58 0

2        52      52 0


SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;


Database altered.


SQL> @d


NAME   INSTANCE_NAME    DB_UNIQUE_NAME   OPEN_MODE        DATABASE_ROLE CURRENT_SCN    FAILOVER_SCN

--------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

RAC   racdr    racdr   MOUNTED        PRIMARY YES    3788307


  • Shutdown new primary database and start database in normal state

SQL> SHUTDOWN IMMEDIATE;

ORA-01109: database not open


Database dismounted.

ORACLE instance shut down.

SQL> startup

ORACLE instance started.


Total System Global Area 2432696320 bytes

Fixed Size     8795664 bytes

Variable Size   738200048 bytes

Database Buffers 1677721600 bytes

Redo Buffers     7979008 bytes

Database mounted.

Database opened.

SQL> @d


NAME   INSTANCE_NAME    DB_UNIQUE_NAME   OPEN_MODE        DATABASE_ROLE CURRENT_SCN    FAILOVER_SCN

--------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

RAC   racdr    racdr   READ WRITE        PRIMARY YES    3788307


  • Forcefully switching  log file for testing purpose so we can confirm MRP0 process running fine on new DR 

SQL> alter system switch logfile;


System altered.


SQL> /


System altered.


SQL> /


System altered.


SQL> @standby


NAME   OPEN_MODE        DATABASE_ROLE

--------- -------------------- ----------------

RAC   READ WRITE        PRIMARY



PROCESS   STATUS SEQUENCE#

--------- ------------ ----------

DGRD   ALLOCATED 0

ARCH   CLOSING        64

DGRD   ALLOCATED 0

ARCH   CLOSING        61

ARCH   CLOSING        62

ARCH   CLOSING        63

LNS   WRITING        65

DGRD   ALLOCATED 0

DGRD   ALLOCATED 0


9 rows selected.


    Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

1        64      64 0

1        64      64 0

2        54      54 0

2        54      54 0


  • Check new standby database and appy mediaa recovery 

SQL> @standby


NAME   OPEN_MODE        DATABASE_ROLE

--------- -------------------- ----------------

RAC   MOUNTED        PHYSICAL STANDBY



PROCESS   STATUS SEQUENCE#

--------- ------------ ----------

ARCH   CONNECTED 0

DGRD   ALLOCATED 0

DGRD   ALLOCATED 0

ARCH   CLOSING        61

ARCH   CONNECTED 0

ARCH   CONNECTED 0

RFS   IDLE        62

RFS   IDLE 0

RFS   IDLE 0

RFS   IDLE 0


10 rows selected.


    Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

 1        61      58   3

 2        54      52   2


  • Apply Media recovery process on new standby database.

SQL> alter database recover managed standby database disconnect from session;


Database altered.


SQL> @standby


NAME   OPEN_MODE        DATABASE_ROLE

--------- -------------------- ----------------

RAC   MOUNTED        PHYSICAL STANDBY


PROCESS   STATUS SEQUENCE#

--------- ------------ ----------

ARCH   CONNECTED 0

DGRD   ALLOCATED 0

DGRD   ALLOCATED 0

ARCH   CLOSING        64

ARCH   CLOSING        62

ARCH   CLOSING        63

RFS   IDLE        65

RFS   IDLE 0

RFS   IDLE 0

RFS   IDLE 0

MRP0   APPLYING_LOG        65


11 rows selected.



    Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

 1        64      64   0

 2        54      54   0


  • New Primary PDB database

SQL> sho pdbs


    CON_ID CON_NAME   OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

2 PDB$SEED   READ ONLY  NO

3 RACPDB   MOUNTED

SQL> alter pluggable database all open;


Pluggable database altered.


SQL> sho pdbs


    CON_ID CON_NAME   OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

2 PDB$SEED   READ ONLY  NO

3 RACPDB   READ WRITE NO


SQL> ALTER PLUGGABLE DATABASE ALL SAVE STATE;


Pluggable database altered.


SQL> !hostname

srv4.example.com


SQL> 


  • New Physical Standby database

 SQL> sho pdbs

    CON_ID CON_NAME   OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED   MOUNTED
3 RACPDB   MOUNTED
SQL> !hostname
srv1.example.com

SQL>