Multitenant Example

Multitenant Dataguard: Create multitenant database on priamry and standby database using FILE_NAME_CONVERT parameter

Multitenant - Performing Flashback for PDB using restorepoint

Multitenant - Enable Flashback on CDB and perform flashback on CDB Common User

Multitenant - Performing Backup and Restore on Pluggable database

Multitenant - Performing RMAN Recovery from non-SYSTEM Root Datafile Loss on CDB

Multitenant - Granting the Privileges as Common or Local on CDB and PDB

Multitenant -Creating the Common and Local Roles on CDB and PDB users

Multitenant- Create Local Users on current PDB

Multitenant - Create Common Users in CDB and PDB

Multitenant - Refreshable PDB Creation from local to remote CDB

Multitenant- Relocating existing PDB from Local CDB to Remote CDB

Multitenant- Creating PDB Using the DBMS_PDB Package from Non-CDB

Multitenant - Create PDB from NON CDB database using DBLINK

Multitenant - Unplugged and Plugged database with Different GUID NO copy CLONE method

Multitenant- PDB OPEN STATE 12.2 release

Multitenant - Creating PDB from seed by mapping names of existing files to new file names destination

Multitenant- Create PDB using db links from remote PDB

Multitenant - Unplugged and Plugged database with same GUID copy method

Multitenant - Drop PDB database

Multitenant- PDB open database commands

Multitenant - Triggers 12.1.0.2.0

Multitenant - Unplugged and Plugged database with Different GUID NO copy CLONE method

Multitenant - Unplugged and Plugged database with Different GUID NO copy method on same host database.

UPLUGGABLE  : PDB2

PLUGGABLE  :PLM3

NOTE: If we  specify CLONE kewords  with NOCOPY then with different GUID  created and able to create PLUGGABLE DATABASE on same host CDB database.


ERROR :
ORA-65122: Pluggable database GUID conflicts with the GUID of an existing container.

Keywards:

• ALTER PLUGGABLE DATABASE PDB2 UNPLUG INTO '/HOME/ORACLE/PDB2.PDB';

• Check compatibilty of unpluggable and pluggable on both host or remote database.

 DECLARE
  2  compatible BOOLEAN := FALSE;
BEGIN
compatible := DBMS_PDB.CHECK_PLUG_COMPATIBILITY(PDB_DESCR_FILE => '/home/oracle/pdb3.xml');
if compatible then
DBMS_OUTPUT.PUT_LINE('It is compatible');
else
DBMS_OUTPUT.PUT_LINE('It is NOT compatible');
end if;
END;
/

• CREATE PLUGGABLE DATABASE PLM3 AS CLONE USING '/HOME/ORACLE/PDB2.PDB' COPY;

EXAMPLE :

SELECT NAME  FROM V$DATAFILE WHERE CON_ID = (SELECT CON_ID FROM V$PDBS WHERE NAME='PDB2');

SQL> sho pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PLM1                           READ WRITE NO
         4 PDB2                           READ WRITE NO
         5 PLM2                           READ WRITE NO
SQL> SELECT NAME  FROM V$DATAFILE WHERE CON_ID = (SELECT CON_ID FROM V$PDBS WHERE NAME='PDB2');

NAME
----------------------------------------------------------------------------------------------------
/u01/app/oracle/oradata/SMRCDB/PDB2/system01.dbf
/u01/app/oracle/oradata/SMRCDB/PDB2/sysaux01.dbf
/u01/app/oracle/oradata/SMRCDB/PDB2/undotbs01.dbf

SQL> ALTER SESSION SET CONTAINER=PDB2;

Session altered.

SQL> SELECT ROUND(SUM(BYTES)/1024/1024,2) SPACE_IN_MB FROM DBA_SEGMENTS;

SPACE_IN_MB
-----------
     585.75

SQL>
ALTER PLUGGABLE DATABASE pdb2 UNPLUG INTO '/home/oracle/pdb2.pdb';


SQL> ALTER SESSION SET CONTAINER=cdb$root;

Session altered.

SQL> ALTER PLUGGABLE DATABASE PDB2 close;

Pluggable database altered.

SQL> ALTER PLUGGABLE DATABASE pdb2 UNPLUG INTO '/home/oracle/pdb2.pdb';

Pluggable database altered.


SQL> host ls -alh /home/oracle/pdb2.pdb
-rw-r--r-- 1 oracle oinstall 161M Aug  7 19:59 /home/oracle/pdb2.pdb

SQL>
CREATE PLUGGABLE DATABASE PLM3 USING '/home/oracle/pdb2.pdb' COPY;

SQL> CREATE PLUGGABLE DATABASE PLM3 USING '/home/oracle/pdb2.pdb' COPY;
CREATE PLUGGABLE DATABASE PLM3 USING '/home/oracle/pdb2.pdb' COPY
*
ERROR at line 1:
ORA-65122: Pluggable database GUID conflicts with the GUID of an existing container.


SQL>
CREATE PLUGGABLE DATABASE pdb6 AS CLONE USING '/home/oracle/pdb5.pdb' COPY;

CREATE PLUGGABLE DATABASE PLM3 AS CLONE USING '/home/oracle/pdb2.pdb' COPY;

SQL> CREATE PLUGGABLE DATABASE PLM3 AS CLONE USING '/home/oracle/pdb2.pdb' COPY;

Pluggable database created.


SQL> ALTER PLUGGABLE DATABASE PLM3 open;

Pluggable database altered.

SQL> col pdb_name format a5
SQL> SELECT PDB_NAME, STATUS FROM CDB_PDBS WHERE PDB_NAME IN ('PDB2','PLM3');

PDB_N STATUS
----- ----------
PDB2  UNPLUGGED
PLM3  NORMAL

SQL> SELECT OPEN_MODE FROM V$PDBS WHERE NAME IN ('PDB2','PLM3');

OPEN_MODE
----------
MOUNTED
READ WRITE


SQL> sho pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PLM1                           READ WRITE NO
         4 PDB2                           MOUNTED
         5 PLM2                           READ WRITE NO
         7 PLM3                           READ WRITE NO
SQL> SELECT GUID FROM V$PDBS WHERE NAME='PLM3';

GUID
--------------------------------
AC4B491CE9DAB663E0534938A8C039B1

SQL>  SELECT GUID FROM V$PDBS WHERE NAME='PDB2';

GUID
--------------------------------
AC217FDB78BE1A68E0534938A8C0C30E

Delete and Re-create MGMTDB database Grid 12R2 12.2.0.1.0 version

Scenario Preview:

We have 12c R2 12.2.0.1.0 two node RAC database. installed on OEL 6.7 operating system on virtual box. This is test database structure. All services started except MGMTDB database . After troubleshooting on logs we found cluseter ware services unable to connect MGTDB database and further found error on redo log file. Since we do not have RMAN backup to resolve this error so we decided to delete and re-create MGMTDB database.  You can follow metalink docs (Doc ID 1589394.1)

MGMTDB log Error :

ORA-00742: Log read detects lost write in thread 1 sequence 10 block 44763
ORA-00312: online log 1 thread 1:
'+CRS/_MGMTDB/ONLINELOG/group_1.264.1043438559'
***********************************************************************
CRS alert log error:

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(CONNECT_TIMEOUT=60)(ADDRESS=(PROTOCOL=tcp)(HOST=srv-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=GIMR_DSCREP_10)(CID=(PROGRAM=ologgerd)(HOST=srv1.example.com)(USER=root)))(SECURITY=(ENCRYPTION_CLIENT=requested)))

  VERSION INFORMATION:
        TNS for Linux: Version 12.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
  Time: 02-AUG-2020 00:50:07
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

***********************************************************************
MGMTB Instance Error:

[grid@srv1 root]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@srv1 root]$ export ORACLE_SID=-MGMTDB
[grid@srv1 root]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sun Aug 2 00:18:29 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  8801008 bytes
Variable Size             436208912 bytes
Database Buffers          624951296 bytes
Redo Buffers                3780608 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread 1 sequence 10 block 44763
ORA-00312: online log 1 thread 1:
'+CRS/_MGMTDB/ONLINELOG/group_1.264.1043438559'

MGMTLSNR Listener status :

[root@srv1 trace]# lsnrctl status MGMTLSNR

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 02-AUG-2020 00:17:22

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MGMTLSNR)))

STATUS of the LISTENER

------------------------

Alias                     MGMTLSNR

Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production

Start Date                01-AUG-2020 23:24:19

Uptime                    0 days 0 hr. 53 min. 3 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora

Listener Log File         /u01/app/grid/diag/tnslsnr/srv1/mgmtlsnr/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MGMTLSNR)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.10.1)(PORT=1526)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.224.241)(PORT=1526)))

The listener supports no services

The command completed successfully

[grid@srv1 _mgmtdb]$ crsctl status resource -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ASMIBNETLSNR_ASM.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.CRS.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.DATA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.FRA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  OFFLINE      srv2                     STABLE

ora.chad

               ONLINE  OFFLINE      srv1                     STABLE

               ONLINE  OFFLINE      srv2                     STABLE

ora.net1.network

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.ons

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.proxy_advm

               OFFLINE OFFLINE      srv1                     STABLE

               OFFLINE OFFLINE      srv2                     STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       srv1                     STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       srv2                     STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.

                                                             168.10.1,STABLE

ora.asm

      1        ONLINE  ONLINE       srv1                     Started,STABLE

      2        ONLINE  ONLINE       srv2                     Started,STABLE

ora.cvu

      1        ONLINE  ONLINE       srv2                     STABLE

ora.mgmtdb

      1        ONLINE  INTERMEDIATE srv1                     Mounted (Closed),Ope

                                                             n Initiated,STABLE

ora.qosmserver

      1        ONLINE  ONLINE       srv2                     STABLE

ora.rac.db

      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       srv2                     STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       srv2                     STABLE

ora.srv1.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.srv2.vip

      1        ONLINE  ONLINE       srv2                     STABLE

--------------------------------------------------------------------------------

[grid@srv1 _mgmtdb]$

Delelete MGMTDB steps:
Stop and disable ora.crf resource so it will not start automatically.

[root@srv1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
[root@srv1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@srv2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

Delete MGMTDB (grid user):

(since mgmtdb was running on another node srv2 and mgmtlsnr is running on srv1 so we got error below this is just example error)

Procedure to run mgmtdb on node srv1 (optional u can skip  if you do not face error below):

[grid@srv1 bin]$ ./dbca -silent -deleteDatabase -sourceDB -MGMTDB
[FATAL] [DBT-10003] Delete operation for Oracle Grid Infrastructure Management Repository (GIMR) cannot be performed on the current node (srv1).
   CAUSE: Oracle GIMR is running on a remote node (srv2).
   ACTION: Invoke DBCA on the remote node (srv2) to delete Oracle GIMR.

srvctl stop mgmtdb (run on srv2 node with grid user)

srvctl start mgmtdb -n srv1 ( run on srv1 node with  grid user)

sqlplus  / as sysbda

startup mount;

Stop and disable ora.crf resource on each node so it will not start automatically on.

[root@srv1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
[root@srv1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@srv2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

Delete command step:

[grid@srv1 bin]$ ./dbca -silent -deleteDatabase -sourceDB -MGMTDB

Connecting to database

4% complete

9% complete

14% complete

19% complete

23% complete

28% complete

47% complete

Updating network configuration files

52% complete

Deleting instance and datafiles

76% complete

100% complete

Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log" for further details.

[grid@srv1 bin]$ cat "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log"

[ 2020-08-02 19:55:46.748 IST ] The Database Configuration Assistant will delete the Oracle instance and datafiles for your database. All information in the database will be destroyed. Do you want to proceed?

[ 2020-08-02 19:55:46.914 IST ] Connecting to database

DBCA_PROGRESS : 4%

DBCA_PROGRESS : 9%

DBCA_PROGRESS : 14%

DBCA_PROGRESS : 19%

DBCA_PROGRESS : 23%

DBCA_PROGRESS : 28%

DBCA_PROGRESS : 47%

[ 2020-08-02 19:57:00.721 IST ] Updating network configuration files

DBCA_PROGRESS : 52%

[ 2020-08-02 19:57:00.790 IST ] Deleting instance and datafiles

DBCA_PROGRESS : 76%

DBCA_PROGRESS : 100%

[ 2020-08-02 19:57:47.860 IST ] Database deletion completed.

Delete MGMTDB at OS level Also:

[grid@srv1 bin]$
ASMCMD> cd _MGMTDB/
ASMCMD> ls
FD9AC0F7C36E4438E043B6A9E80A24D5/
FD9B43BF6A646F8CE043B6A9E80A2815/
ASMCMD> pwd
+CRS/_MGMTDB
ASMCMD> rm -rf *
ASMCMD>
[ 2020-08-02 19:57:47.860 IST ] Database deletion completed.

Create MGMTDB database command:

[grid@srv1 bin]$ ./dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
6% complete
7% complete
22% complete
36% complete
Creating and starting Oracle instance
37% complete
39% complete
43% complete
44% complete
45% complete
49% complete
53% complete
54% complete
55% complete
58% complete
Completing Database Creation
59% complete
62% complete
65% complete
68% complete
Executing Post Configuration Actions
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details.

In case PDB creation command in our case it is non-cdb rac 12R2 so we skip it:

 /u01/app/12.2.0/grid/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB \
-pdbName rac -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb \
-PDBMetadataFile /u01/app/12.2.0/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true

Start and enable ora.crf resource on each node so it will start automatically.

[root@srv1 bin]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@srv1 bin]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'srv1'

CRS-2676: Start of 'ora.crf' on 'srv1' succeeded

[root@srv2 bin]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@srv2 bin]# crsctl start res ora.crf -init

CRS-2672: Attempting to start 'ora.crf' on 'srv2'

CRS-2676: Start of 'ora.crf' on 'srv2' succeeded

[root@srv2 bin]#

Check the status of CRS resource:

grid@srv1 bin]$[root@srv1 bin]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMIBNETLSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  OFFLINE      srv2                     STABLE
ora.chad
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.proxy_advm
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.
                                                             168.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

We just Stop and start clusterware service again (optional):

[root@srv1 bin]# crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'srv2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv2'

CRS-2673: Attempting to stop 'ora.rac.db' on 'srv2'

CRS-2673: Attempting to stop 'ora.qosmserver' on 'srv2'

CRS-2673: Attempting to stop 'ora.crsd' on 'srv1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv1'

CRS-2673: Attempting to stop 'ora.mgmtdb' on 'srv1'

CRS-2673: Attempting to stop 'ora.rac.db' on 'srv1'

CRS-2677: Stop of 'ora.qosmserver' on 'srv2' succeeded

CRS-2677: Stop of 'ora.mgmtdb' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'srv1'

CRS-2677: Stop of 'ora.rac.db' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv2'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv2'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'srv2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'srv2'

CRS-2673: Attempting to stop 'ora.cvu' on 'srv2'

CRS-2673: Attempting to stop 'ora.srv2.vip' on 'srv2'

CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.scan2.vip' on 'srv2'

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on 'srv2'

CRS-2677: Stop of 'ora.DATA.dg' on 'srv2' succeeded

CRS-2677: Stop of 'ora.CRS.dg' on 'srv2' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv2'

CRS-2677: Stop of 'ora.MGMTLSNR' on 'srv1' succeeded

CRS-2677: Stop of 'ora.scan2.vip' on 'srv2' succeeded

CRS-2677: Stop of 'ora.srv2.vip' on 'srv2' succeeded

CRS-2677: Stop of 'ora.scan3.vip' on 'srv2' succeeded

CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv2'

CRS-2677: Stop of 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv2' succeeded

CRS-2677: Stop of 'ora.cvu' on 'srv2' succeeded

CRS-2677: Stop of 'ora.rac.db' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv1'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv1'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'srv1'

CRS-2677: Stop of 'ora.CRS.dg' on 'srv1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'srv1' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'srv1'

CRS-2673: Attempting to stop 'ora.srv1.vip' on 'srv1'

CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv1'

CRS-2677: Stop of 'ora.scan1.vip' on 'srv1' succeeded

CRS-2677: Stop of 'ora.srv1.vip' on 'srv1' succeeded

CRS-2677: Stop of 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'srv1'

CRS-2677: Stop of 'ora.ons' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'srv1'

CRS-2677: Stop of 'ora.net1.network' on 'srv1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv1' has completed

CRS-2673: Attempting to stop 'ora.ons' on 'srv2'

CRS-2677: Stop of 'ora.ons' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'srv2'

CRS-2677: Stop of 'ora.net1.network' on 'srv2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv2' has completed

CRS-2677: Stop of 'ora.crsd' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'srv1'

CRS-2673: Attempting to stop 'ora.evmd' on 'srv1'

CRS-2673: Attempting to stop 'ora.storage' on 'srv1'

CRS-2677: Stop of 'ora.crsd' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'srv2'

CRS-2673: Attempting to stop 'ora.evmd' on 'srv2'

CRS-2673: Attempting to stop 'ora.storage' on 'srv2'

CRS-2677: Stop of 'ora.evmd' on 'srv2' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'srv2' succeeded

CRS-2677: Stop of 'ora.storage' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv2'

CRS-2677: Stop of 'ora.evmd' on 'srv1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'srv1' succeeded

CRS-2677: Stop of 'ora.storage' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv1'

CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'srv1'

CRS-2677: Stop of 'ora.cssd' on 'srv1' succeeded

CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'srv2'

CRS-2677: Stop of 'ora.cssd' on 'srv2' succeeded

[root@srv1 bin]# crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv2'

CRS-2672: Attempting to start 'ora.evmd' on 'srv2'

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv1'

CRS-2672: Attempting to start 'ora.evmd' on 'srv1'

CRS-2676: Start of 'ora.cssdmonitor' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'srv1'

CRS-2672: Attempting to start 'ora.diskmon' on 'srv1'

CRS-2676: Start of 'ora.evmd' on 'srv1' succeeded

CRS-2676: Start of 'ora.diskmon' on 'srv1' succeeded

CRS-2676: Start of 'ora.evmd' on 'srv2' succeeded

CRS-2676: Start of 'ora.cssdmonitor' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'srv2'

CRS-2672: Attempting to start 'ora.diskmon' on 'srv2'

CRS-2676: Start of 'ora.diskmon' on 'srv2' succeeded

CRS-2676: Start of 'ora.cssd' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'srv1'

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv1'

CRS-2676: Start of 'ora.cssd' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'srv2'

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv2'

CRS-2676: Start of 'ora.ctssd' on 'srv1' succeeded

CRS-2676: Start of 'ora.ctssd' on 'srv2' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'srv1'

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'srv2'

CRS-2676: Start of 'ora.asm' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'srv2'

CRS-2676: Start of 'ora.asm' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'srv1'

CRS-2676: Start of 'ora.storage' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'srv1'

CRS-2676: Start of 'ora.crsd' on 'srv1' succeeded

CRS-2676: Start of 'ora.storage' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'srv2'

CRS-2676: Start of 'ora.crsd' on 'srv2' succeeded

[root@srv1 bin]# ps -ef |grep pmon
grid     11035     1  0 20:50 ?        00:00:00 asm_pmon_+ASM1
grid     11940     1  0 20:51 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   12166     1  0 20:51 ?        00:00:00 ora_pmon_rac1
root     19306 10230  0 20:59 pts/0    00:00:00 grep pmon
[root@srv1 bin]# ps -ef |grep lsnr
grid     11539     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr ASMIBNETLSNR_ASM -no_crs_notify -inherit
grid     11548     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
grid     11660     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid     11679     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid     11682     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     19425 11354  0 20:59 ?        00:00:00 /u01/app/12.2.0/grid/bin/lsnrctl status ASMIBNETLSNR_ASM
root     19438 10230  0 20:59 pts/0    00:00:00 grep lsnr

[oracle@srv2 bin]$ ps -ef |grep pmon

grid     12764     1  0 20:58 ?        00:00:00 asm_pmon_+ASM2

oracle   13686     1  0 20:59 ?        00:00:00 ora_pmon_rac2

oracle   14068  3329  0 20:59 pts/0    00:00:00 grep pmon

[oracle@srv2 bin]$ ps -ef |grep lsnr

grid     12022     1  0 20:57 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr ASMIBNETLSNR_ASM -no_crs_notify -inherit

grid     12309     1  0 20:58 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit

oracle   14132  3329  0 20:59 pts/0    00:00:00 grep lsnr

[root@srv1 bin]# crsctl status resource -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ASMIBNETLSNR_ASM.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.CRS.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.DATA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.FRA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  OFFLINE      srv2                     STABLE

ora.chad

               OFFLINE OFFLINE      srv1                     STABLE

               OFFLINE OFFLINE      srv2                     STABLE

ora.net1.network

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.ons

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.proxy_advm

               OFFLINE OFFLINE      srv1                     STABLE

               OFFLINE OFFLINE      srv2                     STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       srv1                     STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       srv1                     STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.

                                                             168.10.1,STABLE

ora.asm

      1        ONLINE  ONLINE       srv1                     Started,STABLE

      2        ONLINE  ONLINE       srv2                     Started,STABLE

ora.cvu

      1        ONLINE  ONLINE       srv1                     STABLE

ora.mgmtdb

      1        ONLINE  ONLINE       srv1                     Open,STABLE

ora.qosmserver

      1        ONLINE  ONLINE       srv1                     STABLE

ora.rac.db

      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       srv2                     STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.srv1.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.srv2.vip

      1        ONLINE  ONLINE       srv2                     STABLE

--------------------------------------------------------------------------------

[root@srv1 bin]#