Saturday 20 November 2021

ggsci - DML replication from source GG to Target GG on pluggable database 19c

 DML replication from source GG to Target GG on pluggable database 19c

Container source database : sourcegg1
source pdb name : sgg1

Container Target database : targetgg1 
target pdb name : tgg1 

Golden gate version : 19c linux on both source and target 

oracle database version : 19c on both source and target 

source container schema name : c##ggsource

target container schema name : c##ggtarget

Source ggci scapture process exint and datapump process is dpint 

Target ggsci replicate process repint


Source hostname: srv6.example.com 

Target hostname : srv7.example.com


  •  Enable replication parameter on source database sourcegg1

SQL> alter system set enable_goldengate_replication=TRUE;

  • Create common user on source contianer database 

System altered.

SQL> create user c##ggsource identified by ggsource default tablespace users;


User created.

  • Add supplemental log data columns to capture information from redo log 

SQL> alter database add supplemental log data (all) columns;

  • Grant dba privilege to all pdb or we can provide to specific pdb as well

Database altered.

SQL> grant dba to c##ggsource container=all;

Grant succeeded.
  • Enable archive log on both source and target database in our case it is enabled
SQL> select log_mode from v$database;

LOG_MODE
------------
ARCHIVELOG
  • Connect PDB database 

SQL> alter session set container=SGG1;

Session altered.

SQL> sho pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 SGG1                           READ WRITE NO

  • Create table supplier for dml replication as an example and insert value later we will do further insert that will reflect on target database
SQL> CREATE TABLE hr.supplier(supplier_id numeric(10) not null,supplier_name varchar2(50) not null,contact_name varchar2(50),CONSTRAINT supplier_pk PRIMARY KEY (supplier_id));

SQL> INSERT INTO hr.supplier values (1,'ABCD1','ABCD1');

1 row created.


SQL> INSERT INTO hr.supplier values (2,'ABCD2','ABCD2');

1 row created.

SQL> commit;

Commit complete.


SQL>  select * from hr.supplier;

SUPPLIER_ID SUPPLIER_NAME
----------- --------------------------------------------------
CONTACT_NAME
--------------------------------------------------
          1 ABCD1
ABCD1

          2 ABCD2
ABCD2



  • Target container database which is targetgg1 and pdb is TGG1


SQL> sho pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 TGG1                           MOUNTED


SQL> alter pluggable database all open;

Pluggable database altered.

SQL> ALTER PLUGGABLE DATABASE ALL SAVE STATE;

Pluggable database altered.
  • Enable golden gate replication parameter true on container database 
SQL> alter system set enable_goldengate_replication=TRUE;

System altered.
  • Create common user and grant dba privilege to all pdb or we can give privilege to specific pdb as well

SQL> grant dba to c##ggtarget container=all;

Grant succeeded.


  • Connect to source GG and create extract and datapump values for replication 
[oracle@srv6 ~]$ cd $GG_HOME
[oracle@srv6 sourcegg1]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.



GGSCI (srv6.example.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED

  • Start manager  process
GGSCI (srv6.example.com) 2> start mgr
Manager started.
  • Connect pdb with common user 
GGSCI (srv6.example.com) 1> dblogin userid c##ggsource@sgg1,password ggsource
Successfully logged into database SGG1.

  • Add trandata for table replication or mention schema * in terms of al table  

GGSCI (srv6.example.com as c##ggsource@sourcegg1/SGG1) 3> add trandata sgg1.hr.supplier

2021-11-19 22:29:01  INFO    OGG-15132  Logging of supplemental redo data enabled for table SGG1.HR.SUPPLIER.

2021-11-19 22:29:01  INFO    OGG-15133  TRANDATA for scheduling columns has been added on table SGG1.HR.SUPPLIER.

2021-11-19 22:29:01  INFO    OGG-15135  TRANDATA for instantiation CSN has been added on table SGG1.HR.SUPPLIER.

2021-11-19 22:29:02  INFO    OGG-10471  ***** Oracle Goldengate support information on table HR.SUPPLIER *****
Oracle Goldengate support native capture on table HR.SUPPLIER.
Oracle Goldengate marked following column as key columns on table HR.SUPPLIER: SUPPLIER_ID.
  • Login with container db common user 
GGSCI (srv6.example.com as c##ggsource@sourcegg1/SGG1) 4> dblogin userid c##ggsource,password ggsource
Successfully logged into database CDB$ROOT.
  • edit  manager with port values 7810 information 
GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 5> edit param mgr

  • Add extract details for capturing commit dml details

GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 6> ADD EXTRACT extint, INTEGRATED TRANLOG, BEGIN NOW
EXTRACT (Integrated) added.
  • Register capture extract with pdb ssg1

GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 7> REGISTER EXTRACT EXTINT DATABASE CONTAINER (sgg1);

2021-11-19 22:33:35  INFO    OGG-02003  Extract EXTINT successfully registered with database at SCN 2313961.


GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 10> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     STOPPED     EXTINT      00:00:00      00:03:23


  • Connect  contianer db add trail  location connecting with  capture extint process

[oracle@srv6 sourcegg1]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.


GGSCI (srv6.example.com) 1> dblogin userid c##ggsource,password ggsource
Successfully logged into database CDB$ROOT.


GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 2> ADD EXTTRAIL  /u01/app/oracle/product/sourcegg1/dirdat/lt, EXTRACT extint
EXTTRAIL added.
  • Add datapump process to transfer trail file to remote trail file location 
GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 3> ADD EXTRACT dpint, EXTTRAILSOURCE /u01/app/oracle/product/sourcegg1/dirdat/lt
EXTRACT added.

  • Add and view extint parameter process 
GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 4> EDIT PARAMS extint


GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 9> view param extint

EXTRACT extint
USERID c##ggsource, PASSWORD ggsource   --connection to CDB$ROOT database
TRANLOGOPTIONS INTEGRATEDPARAMS (MAX_SGA_SIZE 100)
EXTTRAIL /u01/app/oracle/product/sourcegg1/dirdat/lt
TABLE sgg1.hr.supplier;


  • Add and view dpint parameter process

GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 6> EDIT PARAMS dpint


GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 8> view param dpint

EXTRACT dpint
USERID c##ggsource, PASSWORD ggsource
RMTHOST 192.168.1.130, MGRPORT 7810
RMTTRAIL /u01/app/oracle/product/targetgg1/dirdat/rt
PASSTHRU
TABLE sgg1.hr.supplier;
  • Add remote trail location using container db login 
GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 5> ADD RMTTRAIL /u01/app/oracle/product/targetgg1/dirdat/rt, EXTRACT dpint
RMTTRAIL added.

  • Start capture and datapump process extint and dpint 
GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 11> start EXTINT

Sending START request to MANAGER ...
EXTRACT EXTINT starting


GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 12> start DPINT

Sending START request to MANAGER ...
EXTRACT DPINT starting

  • View status of process in ggsci console
GGSCI (srv6.example.com as c##ggsource@sourcegg1/CDB$ROOT) 13> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPINT       00:00:00      00:12:27
EXTRACT     RUNNING     EXTINT      00:00:00      00:23:07


  • Connect to target ggsci console 

[oracle@srv7 targetgg1]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.



GGSCI (srv7.example.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


GGSCI (srv7.example.com) 2> start mgr
Manager started.


GGSCI (srv7.example.com) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
  • Add port on manager process 
GGSCI (srv7.example.com) 24> view param mgr

PORT 7810


GGSCI (srv7.example.com) 25>
  • Add replication process on target ggsci console

GGSCI (srv7.example.com) 4> ADD REPLICAT repint,integrated EXTTRAIL /u01/app/oracle/product/targetgg1/dirdat/rt
REPLICAT (Integrated) added.
  • Edit and view replicate parameters 

GGSCI (srv7.example.com) 5> EDIT PARAMS repint


GGSCI (srv7.example.com) 8> view param repint

REPLICAT repint
USERID c##ggtarget@TGG1, PASSWORD ggtarget –Connection to pluggable
DBOPTIONS INTEGRATEDPARAMS(parallelism 4)
ASSUMETARGETDEFS
MAP sgg1.hr.supplier, TARGET tgg1.hr.supplier;

  • Connect to source pluggable  database SGG1  and insert the values , It should reflect on target database TGG1
SQL> sho pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 SGG1                           READ WRITE NO
SQL>


SQL> INSERT INTO hr.supplier values (5,'ABCD5','ABCD5');

1 row created.

SQL> commit;

Commit complete.

ggserr.log

2021-11-20T12:32:32.763+0530  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.
2021-11-20T13:50:24.420+0530  INFO    OGG-06507  Oracle GoldenGate Capture for Oracle, extint.prm:  MAP (TABLE) resolved (entry sgg1.hr.supplier): TABLE "SGG1"."HR"."SUPPLIER".
2021-11-20T13:50:24.574+0530  INFO    OGG-06509  Oracle GoldenGate Capture for Oracle, extint.prm:  Using the following key columns for source table SGG1.HR.SUPPLIER: SUPPLIER_ID.
2021-11-20T13:50:24.708+0530  INFO    OGG-06507  Oracle GoldenGate Capture for Oracle, extint.prm:  MAP (TABLE) resolved (entry sgg1.hr.supplier): TABLE "SGG1"."HR"."SUPPLIER".
2021-11-20T13:50:24.708+0530  INFO    OGG-06509  Oracle GoldenGate Capture for Oracle, extint.prm:  Using the following key columns for source table SGG1.HR.SUPPLIER: SUPPLIER_ID.
2021-11-20T13:50:26.217+0530  INFO    OGG-02263  Oracle GoldenGate Capture for Oracle, dpint.prm:  Passthru MAP (TABLE) resolved (entry sgg1.hr.supplier): TABLE "SGG1"."HR"."SUPPLIER".


  • After inserting table values reflected on target pluggable database TGG1 


SQL> sho pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 TGG1                           READ WRITE NO


SQL> select * from hr.supplier order by SUPPLIER_ID asc;

SUPPLIER_ID SUPPLIER_NAME                            CONTACT_NAME
----------- ---------------------------------------- --------------------------------------------------
          1 ABCD1                                    ABCD1
          2 ABCD2                                    ABCD2
          3 ABCD3                                    ABCD3
          4 ABCD4                                    ABCD4
          5 ABCD5                                    ABCD5


ggserr.log 

2021-11-20T12:32:46.523+0530  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.
2021-11-20T13:50:29.073+0530  INFO    OGG-06505  Oracle GoldenGate Delivery for Oracle, repint.prm:  MAP resolved (entry sgg1.hr.supplier): MAP "SGG1"."HR"."SUPPLIER", TARGET tgg1.hr.supplier.
2021-11-20T13:50:35.176+0530  INFO    OGG-02756  Oracle GoldenGate Delivery for Oracle, repint.prm:  The definition for table SGG1.HR.SUPPLIER is obtained from the trail file.
2021-11-20T13:50:35.176+0530  INFO    OGG-06511  Oracle GoldenGate Delivery for Oracle, repint.prm:  Using following columns in default map by name: SUPPLIER_ID, SUPPLIER_NAME, CONTACT_NAME.
2021-11-20T13:50:35.177+0530  INFO    OGG-06510  Oracle GoldenGate Delivery for Oracle, repint.prm:  Using the following key columns for target table TGG1.HR.SUPPLIER: SUPPLIER_ID.





Monday 15 November 2021

ggsci - ./ggsci: error while loading shared libraries: libnnz19.so: cannot open shared object file: No such file or directory

  •  Scenario: Getting error after installing GG 19c first time linux 64 machine 

[oracle@srv3 sourcegg1]$ ./ggsci

./ggsci: error while loading shared libraries: libnnz19.so: cannot open shared object file: No such file or directory

  • Solution :  Add GG_HOME and  LD_LIBRARY_PATH on bash_profile 

[oracle@srv3 ~]$ vi .bash_profile

# Get the aliases and functions


if [ -f ~/.bashrc ]; then


        . ~/.bashrc


fi


umask 022


export ORACLE_BASE=/u01/app/oracle


export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1


export ORACLE_HOME_LISTNER=$ORACLE_HOME/network/admin


export TNS_ADMIN=$ORACLE_HOME/network/admin


#$LD_LIBRARY_PATH


GG_HOME=/u01/app/oracle/product/sourcegg1; export GG_HOME


export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$GG_HOME


export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH


export PATH=$PATH:$ORACLE_HOME/bin


export ORACLE_SID=smrrac1


#NLS_DATE_FORMAT=Mon DD YYYY HH24:MI:SS; export NLS_DATE_FORMAT


ORATAB=/etc/oratab;export ORATAB

[oracle@srv3 ~]$

  • Execute changes in bash_profile and start ggsci again 

[oracle@srv3 ~]$ source ~/.bash_profile

[oracle@srv3 ~]$ cd $GG_HOME


[oracle@srv3 sourcegg1]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO

Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29

Operating system character set identified as UTF-8.


Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.


GGSCI (srv3.example.com) 1>



GGSCI (srv3.example.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING


GGSCI (srv3.example.com) 2>

GGSCI - Golden gate 19c Installation on linux machine

 GGSCI - Golden gate 19c Installation on linux machine  

Installation with manager options

  • [oracle@srv3 Disk1]$ ./runInstaller 

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 3797 MB    Passed

Checking swap space: must be greater than 150 MB.   Actual 8191 MB    Passed

Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2021-11-10_04-47-34PM. Please wait ...[oracle@srv3 Disk1]$ 










  • Add gg_home and ld_library on bash_profile

[oracle@srv3 ~]$ source ~/.bash_profile

[oracle@srv3 ~]$ cd $GG_HOME

[oracle@srv3 sourcegg1]$ ./ggsci


Oracle GoldenGate Command Interpreter for Oracle

Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO

Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29

Operating system character set identified as UTF-8.


Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.


GGSCI (srv3.example.com) 1>



GGSCI (srv3.example.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING


GGSCI (srv3.example.com) 2>


  • Installation of golden gate without  manager prcocess 


[oracle@srv4 ~]$ cd fbo_ggs_Linux_x64_shiphome/
[oracle@srv4 fbo_ggs_Linux_x64_shiphome]$ ll
total 4
drwxrwxr-x. 5 oracle oinstall 4096 Oct 18  2019 Disk1
[oracle@srv4 fbo_ggs_Linux_x64_shiphome]$ cd Disk1/
[oracle@srv4 Disk1]$ ll
total 16
drwxrwxr-x.  4 oracle oinstall 4096 Oct 18  2019 install
drwxrwxr-x.  2 oracle oinstall 4096 Oct 18  2019 response
-rwxrwxr-x.  1 oracle oinstall  918 Oct 18  2019 runInstaller
drwxrwxr-x. 12 oracle oinstall 4096 Oct 18  2019 stage
[oracle@srv4 Disk1]$ ./runInstaller 
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 3367 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8184 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2021-11-11_04-56-53PM. Please wait ...[oracle@srv4 Disk1]$ 










  • Set golden gate home in bash_profile

[oracle@srv4 ~]$ source ~./bash_profile
bash: ~./bash_profile: No such file or directory
[oracle@srv4 ~]$ source ~/.bash_profile
[oracle@srv4 ~]$ cd $GG_HOME
[oracle@srv4 targetgg1]$ pwd
/u01/app/oracle/product/targetgg1
[oracle@srv4 targetgg1]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.


GGSCI (srv4.example.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


  • Create sub directories

GGSCI (srv4.example.com) 2> create subdirs

Creating subdirectories under current directory /u01/app/oracle/product/targetgg1

Parameter file                 /u01/app/oracle/product/targetgg1/dirprm: created.
Report file                    /u01/app/oracle/product/targetgg1/dirrpt: created.
Checkpoint file                /u01/app/oracle/product/targetgg1/dirchk: created.
Process status files           /u01/app/oracle/product/targetgg1/dirpcs: created.
SQL script files               /u01/app/oracle/product/targetgg1/dirsql: created.
Database definitions files     /u01/app/oracle/product/targetgg1/dirdef: created.
Extract data files             /u01/app/oracle/product/targetgg1/dirdat: created.
Temporary files                /u01/app/oracle/product/targetgg1/dirtmp: created.
Credential store files         /u01/app/oracle/product/targetgg1/dircrd: created.
Masterkey wallet files         /u01/app/oracle/product/targetgg1/dirwlt: created.
Dump files                     /u01/app/oracle/product/targetgg1/dirdmp: created.


GGSCI (srv4.example.com) 3>

[oracle@srv4 targetgg1]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.

  • Set manager process information 

GGSCI (srv4.example.com) 1> start mgr
ERROR: Parameter file /u01/app/oracle/product/targetgg1/dirprm/mgr.prm does not exist.



GGSCI (srv4.example.com) 6> edit param mgr

with add the port number : 

port 8080


GGSCI (srv4.example.com) 7> start mgr
Manager started.


GGSCI (srv4.example.com) 8> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING


GGSCI (srv4.example.com) 9>


[oracle@srv4 dirprm]$ pwd
/u01/app/oracle/product/targetgg1/dirprm
[oracle@srv4 dirprm]$

[oracle@srv4 targetgg1]$ cd dirprm
[oracle@srv4 dirprm]$ ll
total 4
-rw-r-----. 1 oracle oinstall 11 Nov 11 17:20 mgr.prm
[oracle@srv4 dirprm]$ cat mgr.prm
PORT 8080

[oracle@srv4 dirprm]$


  • login source GG server   srv6 using ggsci


GGSCI (srv6.example.com) 9> dblogin userid c##ggsource@gg1,password ggsource

Successfully logged into database GG1.


GGSCI (srv6.example.com as c##ggsource@sourcegg1/GG1) 10>


  • login target GG Server srv7 using ggsci

GGSCI (srv7.example.com) 5>  dblogin userid c##ggtarget@ggt,password ggtarget
Successfully logged into database GGT.

GGSCI (srv7.example.com as c##ggtarget@targetgg/GGT) 6>


Monday 8 November 2021

RAC - Replace or Recover CSS votedisk in case of failure without backup

 

RAC - Replace or Recover CSS votedisk in case of failure with out backup




 
Scenario: 

We have 3 node 12.2.0.1.0 RAC  setup .  We are running only 2 node  later after voteing disk replace we will start 3rd node as well. Voting disk and OCR is stored in same ASM diskgroup location which is +CRS. Our scenario is in case  repair or replace  votedisk to another disk group without backup.  

step 1 : Stop cluster and crs on all nodes 

crsctl stop crs -f  (on all nodes )

crsctl disable crs  (disable auto start )



step 2 :  Start crs in exclusive mode on one of node 
crsctl start crs -excl 
crsctl start crs -excl -nocrs 



step 3 :  Replace current votedisk  

crsctl replace votedisk +backup

crsctl query css votedisk 


step 4 : stop crs  on current node 

crsctl stop crs

step 5 : start crs and cluster on all the nodes 

crsctl start crs 

crsctl start cluster -all

step 6: check  votedisk location 
ocrcheck

crsctl query css votedisk 

step 7 : verify  cluster on all nodes 

  • Check current location of css votedisk and OCR , In our case currently it is stored in +CRS disk group

[root@srv2 ~]# crsctl  query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

[root@srv2 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   89ccb666fba84f48bffd99f68cc37d9a (/dev/oracleasm/disks/CRS) [CRS]
Located 1 voting disk(s).
[root@srv2 ~]# 


[root@srv2 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          4
Total space (kbytes)     :     409568
Used space (kbytes)      :       2268
Available space (kbytes) :     407300
ID                       :  817426981
Device/File Name         :       +CRS
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

  • Current ASM disk name  first we create backup diskgroup with external redundancy . This is for preparation of replacing votedisk or failure, As we can't replace with same diskgroup .

[root@srv1 ~]# oracleasm listdisks
BACKUP
CRS
DATA
FRA
[root@srv1 ~]# asmcd
bash: asmcd: command not found...
[root@srv1 ~]# asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     43004     8920                0            8920              0             Y  CRS/
MOUNTED  EXTERN  N         512             512   4096  4194304     20476    20300                0           20300              0             N  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304     20476    15592                0           15592              0             N  FRA/

  • Current disk group information status and backup disk is in provisioned state 
ASMCMD> exit


SQL> set lines 999;

col diskgroup for a10

col diskname for a12

col path for a30

select a.name DiskGroup,b.name DiskName, b.total_mb, (b.total_mb-b.free_mb) Used_MB, b.free_mb,b.path,b.header_status from v$asm_disk b, v$asm_diskgroup a where a.group_number (+) =b.group_number order by b.group_number,b.name;SQL> SQL> SQL> SQL> SQL> SQL> SQL> SQL> 

DISKGROUP  DISKNAME   TOTAL_MB    USED_MB FREE_MB PATH HEADER_STATU
---------- ------------ ---------- ---------- ---------- ------------------------------ ------------
0     0        0 /dev/oracleasm/disks/BACKUP PROVISIONED
CRS    CRS_0000      43004 34084     8920 /dev/oracleasm/disks/CRS MEMBER
DATA    DATA_0000      20476   176    20300 /dev/oracleasm/disks/DATA MEMBER
FRA    FRA_0000      20476 4884    15592 /dev/oracleasm/disks/FRA MEMBER

SQL> 


SQL> set pages 40000 lines 120

col PATH for a30

select DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,PATH FROM V$ASM_DISK;SQL> SQL> SQL> SQL> 

DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE   PATH
----------- ------- ------------ ------- -------- ------------------------------
  0 CLOSED  PROVISIONED  ONLINE  NORMAL   /dev/oracleasm/disks/BACKUP
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/CRS
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/FRA
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/DATA



SQL> sho parameter ASM_DISKSTRING

NAME      TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring      string /dev/oracleasm/disks

  • Create diskgroup backup with external redundancy 
SQL> create diskgroup Backup external redundancy disk '/dev/oracleasm/disks/BACKUP' name backup;

Diskgroup created.

SQL> select DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,PATH FROM V$ASM_DISK;

DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE   PATH
----------- ------- ------------ ------- -------- ------------------------------
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/CRS
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/FRA
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/DATA
  0 CACHED  MEMBER ONLINE  NORMAL   /dev/oracleasm/disks/BACKUP

  • Check ASM diskgroup status 
SQL> select a.name DiskGroup,b.name DiskName, b.total_mb, (b.total_mb-b.free_mb) Used_MB, b.free_mb,b.path,b.header_status from v$asm_disk b, v$asm_diskgroup a where a.group_number (+) =b.group_number order by b.group_number,b.name;

DISKGROUP  DISKNAME   TOTAL_MB    USED_MB FREE_MB PATH HEADER_STATU
---------- ------------ ---------- ---------- ---------- ------------------------------ ------------
CRS    CRS_0000      43004 34084     8920 /dev/oracleasm/disks/CRS MEMBER
DATA    DATA_0000      20476   176    20300 /dev/oracleasm/disks/DATA MEMBER
FRA    FRA_0000      20476 4884    15592 /dev/oracleasm/disks/FRA MEMBER
BACKUP    BACKUP      15359    52    15307 /dev/oracleasm/disks/BACKUP MEMBER

SQL> 


SQL>  select state,name,type from v$asm_diskgroup;

STATE     NAME    TYPE
----------- ------------------------------ ------
MOUNTED     CRS    EXTERN
MOUNTED     DATA    EXTERN
MOUNTED     FRA    EXTERN
MOUNTED     BACKUP    EXTERN

  • Now if any real time scenario if voting disk got crashed or some hardware failure in vote disk stop all cluster services forcefully below. 
[root@srv2 ~]# crsctl stop crs -f 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'srv2'
CRS-2673: Attempting to stop 'ora.crsd' on 'srv2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv2'
CRS-2673: Attempting to stop 'ora.chad' on 'srv2'
CRS-2673: Attempting to stop 'ora.rac.db' on 'srv2'
CRS-2677: Stop of 'ora.rac.db' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'srv2'
CRS-2677: Stop of 'ora.FRA.dg' on 'srv2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'srv2' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.srv2.vip' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'srv2'
CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'srv2'
CRS-2677: Stop of 'ora.scan1.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.srv2.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.chad' on 'srv2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'srv1'
CRS-2672: Attempting to start 'ora.srv2.vip' on 'srv1'
CRS-2676: Start of 'ora.srv2.vip' on 'srv1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'srv1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'srv2'
CRS-2677: Stop of 'ora.ons' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'srv2'
CRS-2677: Stop of 'ora.net1.network' on 'srv2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv2' has completed
CRS-2677: Stop of 'ora.crsd' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'srv2'
CRS-2673: Attempting to stop 'ora.crf' on 'srv2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'srv2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'srv2'
CRS-2677: Stop of 'ora.crf' on 'srv2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.storage' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv2'
CRS-2677: Stop of 'ora.mdnsd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'srv2'
CRS-2673: Attempting to stop 'ora.evmd' on 'srv2'
CRS-2677: Stop of 'ora.ctssd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv2'
CRS-2677: Stop of 'ora.cssd' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'srv2'
CRS-2677: Stop of 'ora.gipcd' on 'srv2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'srv2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

  • Disable auto-start  CRS service in case  for OS reboot 

[root@srv2 ~]# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.

  • Start crs service with exclusive mode with no crs services 


Usage:
  crsctl start crs [-excl [-nocrs | -cssonly]] | [-wait | -waithas | -nowait] | [-noautostart]
     Start OHAS on this server
where
     -excl        Start Oracle Clusterware in exclusive mode
     -nocrs       Start Oracle Clusterware in exclusive mode without starting CRS
     -nowait      Do not wait for OHAS to start
     -wait        Wait until startup is complete and display all progress and status messages
     -waithas     Wait until startup is complete and display OHASD progress and status messages
     -cssonly     Start only CSS
     -noautostart Start only OHAS


[root@srv2 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'srv2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'srv2'
CRS-2676: Start of 'ora.mdnsd' on 'srv2' succeeded
CRS-2676: Start of 'ora.evmd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'srv2'
CRS-2676: Start of 'ora.gpnpd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv2'
CRS-2672: Attempting to start 'ora.gipcd' on 'srv2'
CRS-2676: Start of 'ora.cssdmonitor' on 'srv2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'srv2'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv2'
CRS-2676: Start of 'ora.diskmon' on 'srv2' succeeded
CRS-2676: Start of 'ora.cssd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2672: Attempting to start 'ora.ctssd' on 'srv2'
CRS-2676: Start of 'ora.ctssd' on 'srv2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv2'
CRS-2676: Start of 'ora.asm' on 'srv2' succeeded


[root@srv2 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   89ccb666fba84f48bffd99f68cc37d9a (/dev/oracleasm/disks/CRS) [CRS]
Located 1 voting disk(s).
  • Replace votedisk with new diskgroup. We can place votedisk with same OCR location.

[root@srv2 ~]# crsctl replace votedisk +backup
Successful addition of voting disk 6fc2ec78a9024fc2bf02824b67e7da7c.
Successful deletion of voting disk 89ccb666fba84f48bffd99f68cc37d9a.
Successfully replaced voting disk group with +backup.
CRS-4266: Voting file(s) successfully replaced

  • Stop current Cluster Synchronization Services

[root@srv2 ~]# crsctl stop crs
Unable to communicate with the Cluster Synchronization Services daemon.
  • Start CRS service on all nodes 
[root@srv2 ~]# crsctl start cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Start failed, or completed with errors.

[root@srv2 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@srv2 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6fc2ec78a9024fc2bf02824b67e7da7c (/dev/oracleasm/disks/BACKUP) [BACKUP]
Located 1 voting disk(s).
[root@srv2 ~]# 
  • Check all clusterware services 

[root@srv2 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.BACKUP.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.chad
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv2                     169.254.238.221 192.
                                                             168.10.2,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       srv2                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv2                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv2                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv3.vip
      1        ONLINE  INTERMEDIATE srv2                     FAILED OVER,STABLE
--------------------------------------------------------------------------------

  • Now Start node 3 and check vote disk status 
[root@srv2 ~]# ssh srv3
root@srv3's password: 
Last login: Tue Nov  2 16:13:08 2021 from srv1.example.com
[root@srv3 ~]# . oraenv
ORACLE_SID = [root] ? +ASM3
The Oracle base has been set to /u01/app/grid
[root@srv3 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6fc2ec78a9024fc2bf02824b67e7da7c (/dev/oracleasm/disks/BACKUP) [BACKUP]
Located 1 voting disk(s).
[root@srv3 ~]# olsnodes
srv1
srv2
srv3

  • Check clusrtware status from node 3
[root@srv3 ~]# crsctl check cluster -all
**************************************************************
srv1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
srv2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
srv3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

[root@srv3 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.BACKUP.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.chad
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               OFFLINE OFFLINE      srv3                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
               ONLINE  ONLINE       srv3                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv3                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv3                     169.254.187.26 192.1
                                                             68.10.3,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
      3        ONLINE  ONLINE       srv3                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv3                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv3                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv3                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      3        ONLINE  ONLINE       srv3                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv3                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv3.vip
      1        ONLINE  ONLINE       srv3                     STABLE
--------------------------------------------------------------------------------