Delete and Re-create MGMTDB database Grid 12R2 12.2.0.1.0 version

Scenario Preview:

We have 12c R2 12.2.0.1.0 two node RAC database. installed on OEL 6.7 operating system on virtual box. This is test database structure. All services started except MGMTDB database . After troubleshooting on logs we found cluseter ware services unable to connect MGTDB database and further found error on redo log file. Since we do not have RMAN backup to resolve this error so we decided to delete and re-create MGMTDB database.  You can follow metalink docs (Doc ID 1589394.1)

MGMTDB log Error :

ORA-00742: Log read detects lost write in thread 1 sequence 10 block 44763
ORA-00312: online log 1 thread 1:
'+CRS/_MGMTDB/ONLINELOG/group_1.264.1043438559'
***********************************************************************
CRS alert log error:

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(CONNECT_TIMEOUT=60)(ADDRESS=(PROTOCOL=tcp)(HOST=srv-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=GIMR_DSCREP_10)(CID=(PROGRAM=ologgerd)(HOST=srv1.example.com)(USER=root)))(SECURITY=(ENCRYPTION_CLIENT=requested)))

  VERSION INFORMATION:
        TNS for Linux: Version 12.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
  Time: 02-AUG-2020 00:50:07
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

***********************************************************************
MGMTB Instance Error:

[grid@srv1 root]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@srv1 root]$ export ORACLE_SID=-MGMTDB
[grid@srv1 root]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sun Aug 2 00:18:29 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  8801008 bytes
Variable Size             436208912 bytes
Database Buffers          624951296 bytes
Redo Buffers                3780608 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread 1 sequence 10 block 44763
ORA-00312: online log 1 thread 1:
'+CRS/_MGMTDB/ONLINELOG/group_1.264.1043438559'

MGMTLSNR Listener status :

[root@srv1 trace]# lsnrctl status MGMTLSNR

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 02-AUG-2020 00:17:22

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MGMTLSNR)))

STATUS of the LISTENER

------------------------

Alias                     MGMTLSNR

Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production

Start Date                01-AUG-2020 23:24:19

Uptime                    0 days 0 hr. 53 min. 3 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora

Listener Log File         /u01/app/grid/diag/tnslsnr/srv1/mgmtlsnr/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MGMTLSNR)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.10.1)(PORT=1526)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.224.241)(PORT=1526)))

The listener supports no services

The command completed successfully

[grid@srv1 _mgmtdb]$ crsctl status resource -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ASMIBNETLSNR_ASM.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.CRS.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.DATA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.FRA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  OFFLINE      srv2                     STABLE

ora.chad

               ONLINE  OFFLINE      srv1                     STABLE

               ONLINE  OFFLINE      srv2                     STABLE

ora.net1.network

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.ons

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.proxy_advm

               OFFLINE OFFLINE      srv1                     STABLE

               OFFLINE OFFLINE      srv2                     STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       srv1                     STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       srv2                     STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.

                                                             168.10.1,STABLE

ora.asm

      1        ONLINE  ONLINE       srv1                     Started,STABLE

      2        ONLINE  ONLINE       srv2                     Started,STABLE

ora.cvu

      1        ONLINE  ONLINE       srv2                     STABLE

ora.mgmtdb

      1        ONLINE  INTERMEDIATE srv1                     Mounted (Closed),Ope

                                                             n Initiated,STABLE

ora.qosmserver

      1        ONLINE  ONLINE       srv2                     STABLE

ora.rac.db

      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       srv2                     STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       srv2                     STABLE

ora.srv1.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.srv2.vip

      1        ONLINE  ONLINE       srv2                     STABLE

--------------------------------------------------------------------------------

[grid@srv1 _mgmtdb]$

Delelete MGMTDB steps:
Stop and disable ora.crf resource so it will not start automatically.

[root@srv1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
[root@srv1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@srv2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

Delete MGMTDB (grid user):

(since mgmtdb was running on another node srv2 and mgmtlsnr is running on srv1 so we got error below this is just example error)

Procedure to run mgmtdb on node srv1 (optional u can skip  if you do not face error below):

[grid@srv1 bin]$ ./dbca -silent -deleteDatabase -sourceDB -MGMTDB
[FATAL] [DBT-10003] Delete operation for Oracle Grid Infrastructure Management Repository (GIMR) cannot be performed on the current node (srv1).
   CAUSE: Oracle GIMR is running on a remote node (srv2).
   ACTION: Invoke DBCA on the remote node (srv2) to delete Oracle GIMR.

srvctl stop mgmtdb (run on srv2 node with grid user)

srvctl start mgmtdb -n srv1 ( run on srv1 node with  grid user)

sqlplus  / as sysbda

startup mount;

Stop and disable ora.crf resource on each node so it will not start automatically on.

[root@srv1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
[root@srv1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@srv2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

Delete command step:

[grid@srv1 bin]$ ./dbca -silent -deleteDatabase -sourceDB -MGMTDB

Connecting to database

4% complete

9% complete

14% complete

19% complete

23% complete

28% complete

47% complete

Updating network configuration files

52% complete

Deleting instance and datafiles

76% complete

100% complete

Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log" for further details.

[grid@srv1 bin]$ cat "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log"

[ 2020-08-02 19:55:46.748 IST ] The Database Configuration Assistant will delete the Oracle instance and datafiles for your database. All information in the database will be destroyed. Do you want to proceed?

[ 2020-08-02 19:55:46.914 IST ] Connecting to database

DBCA_PROGRESS : 4%

DBCA_PROGRESS : 9%

DBCA_PROGRESS : 14%

DBCA_PROGRESS : 19%

DBCA_PROGRESS : 23%

DBCA_PROGRESS : 28%

DBCA_PROGRESS : 47%

[ 2020-08-02 19:57:00.721 IST ] Updating network configuration files

DBCA_PROGRESS : 52%

[ 2020-08-02 19:57:00.790 IST ] Deleting instance and datafiles

DBCA_PROGRESS : 76%

DBCA_PROGRESS : 100%

[ 2020-08-02 19:57:47.860 IST ] Database deletion completed.

Delete MGMTDB at OS level Also:

[grid@srv1 bin]$
ASMCMD> cd _MGMTDB/
ASMCMD> ls
FD9AC0F7C36E4438E043B6A9E80A24D5/
FD9B43BF6A646F8CE043B6A9E80A2815/
ASMCMD> pwd
+CRS/_MGMTDB
ASMCMD> rm -rf *
ASMCMD>
[ 2020-08-02 19:57:47.860 IST ] Database deletion completed.

Create MGMTDB database command:

[grid@srv1 bin]$ ./dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
6% complete
7% complete
22% complete
36% complete
Creating and starting Oracle instance
37% complete
39% complete
43% complete
44% complete
45% complete
49% complete
53% complete
54% complete
55% complete
58% complete
Completing Database Creation
59% complete
62% complete
65% complete
68% complete
Executing Post Configuration Actions
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details.

In case PDB creation command in our case it is non-cdb rac 12R2 so we skip it:

 /u01/app/12.2.0/grid/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB \
-pdbName rac -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb \
-PDBMetadataFile /u01/app/12.2.0/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true

Start and enable ora.crf resource on each node so it will start automatically.

[root@srv1 bin]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@srv1 bin]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'srv1'

CRS-2676: Start of 'ora.crf' on 'srv1' succeeded

[root@srv2 bin]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@srv2 bin]# crsctl start res ora.crf -init

CRS-2672: Attempting to start 'ora.crf' on 'srv2'

CRS-2676: Start of 'ora.crf' on 'srv2' succeeded

[root@srv2 bin]#

Check the status of CRS resource:

grid@srv1 bin]$[root@srv1 bin]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMIBNETLSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  OFFLINE      srv2                     STABLE
ora.chad
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.proxy_advm
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.
                                                             168.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

We just Stop and start clusterware service again (optional):

[root@srv1 bin]# crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'srv2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv2'

CRS-2673: Attempting to stop 'ora.rac.db' on 'srv2'

CRS-2673: Attempting to stop 'ora.qosmserver' on 'srv2'

CRS-2673: Attempting to stop 'ora.crsd' on 'srv1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv1'

CRS-2673: Attempting to stop 'ora.mgmtdb' on 'srv1'

CRS-2673: Attempting to stop 'ora.rac.db' on 'srv1'

CRS-2677: Stop of 'ora.qosmserver' on 'srv2' succeeded

CRS-2677: Stop of 'ora.mgmtdb' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'srv1'

CRS-2677: Stop of 'ora.rac.db' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv2'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv2'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'srv2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'srv2'

CRS-2673: Attempting to stop 'ora.cvu' on 'srv2'

CRS-2673: Attempting to stop 'ora.srv2.vip' on 'srv2'

CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.scan2.vip' on 'srv2'

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on 'srv2'

CRS-2677: Stop of 'ora.DATA.dg' on 'srv2' succeeded

CRS-2677: Stop of 'ora.CRS.dg' on 'srv2' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv2'

CRS-2677: Stop of 'ora.MGMTLSNR' on 'srv1' succeeded

CRS-2677: Stop of 'ora.scan2.vip' on 'srv2' succeeded

CRS-2677: Stop of 'ora.srv2.vip' on 'srv2' succeeded

CRS-2677: Stop of 'ora.scan3.vip' on 'srv2' succeeded

CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv2'

CRS-2677: Stop of 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv2' succeeded

CRS-2677: Stop of 'ora.cvu' on 'srv2' succeeded

CRS-2677: Stop of 'ora.rac.db' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv1'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv1'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'srv1'

CRS-2677: Stop of 'ora.CRS.dg' on 'srv1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'srv1' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'srv1'

CRS-2673: Attempting to stop 'ora.srv1.vip' on 'srv1'

CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv1'

CRS-2677: Stop of 'ora.scan1.vip' on 'srv1' succeeded

CRS-2677: Stop of 'ora.srv1.vip' on 'srv1' succeeded

CRS-2677: Stop of 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'srv1'

CRS-2677: Stop of 'ora.ons' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'srv1'

CRS-2677: Stop of 'ora.net1.network' on 'srv1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv1' has completed

CRS-2673: Attempting to stop 'ora.ons' on 'srv2'

CRS-2677: Stop of 'ora.ons' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'srv2'

CRS-2677: Stop of 'ora.net1.network' on 'srv2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv2' has completed

CRS-2677: Stop of 'ora.crsd' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'srv1'

CRS-2673: Attempting to stop 'ora.evmd' on 'srv1'

CRS-2673: Attempting to stop 'ora.storage' on 'srv1'

CRS-2677: Stop of 'ora.crsd' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'srv2'

CRS-2673: Attempting to stop 'ora.evmd' on 'srv2'

CRS-2673: Attempting to stop 'ora.storage' on 'srv2'

CRS-2677: Stop of 'ora.evmd' on 'srv2' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'srv2' succeeded

CRS-2677: Stop of 'ora.storage' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv2'

CRS-2677: Stop of 'ora.evmd' on 'srv1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'srv1' succeeded

CRS-2677: Stop of 'ora.storage' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv1'

CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'srv1'

CRS-2677: Stop of 'ora.cssd' on 'srv1' succeeded

CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'srv2'

CRS-2677: Stop of 'ora.cssd' on 'srv2' succeeded

[root@srv1 bin]# crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv2'

CRS-2672: Attempting to start 'ora.evmd' on 'srv2'

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv1'

CRS-2672: Attempting to start 'ora.evmd' on 'srv1'

CRS-2676: Start of 'ora.cssdmonitor' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'srv1'

CRS-2672: Attempting to start 'ora.diskmon' on 'srv1'

CRS-2676: Start of 'ora.evmd' on 'srv1' succeeded

CRS-2676: Start of 'ora.diskmon' on 'srv1' succeeded

CRS-2676: Start of 'ora.evmd' on 'srv2' succeeded

CRS-2676: Start of 'ora.cssdmonitor' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'srv2'

CRS-2672: Attempting to start 'ora.diskmon' on 'srv2'

CRS-2676: Start of 'ora.diskmon' on 'srv2' succeeded

CRS-2676: Start of 'ora.cssd' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'srv1'

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv1'

CRS-2676: Start of 'ora.cssd' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'srv2'

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv2'

CRS-2676: Start of 'ora.ctssd' on 'srv1' succeeded

CRS-2676: Start of 'ora.ctssd' on 'srv2' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'srv1'

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'srv2'

CRS-2676: Start of 'ora.asm' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'srv2'

CRS-2676: Start of 'ora.asm' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'srv1'

CRS-2676: Start of 'ora.storage' on 'srv1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'srv1'

CRS-2676: Start of 'ora.crsd' on 'srv1' succeeded

CRS-2676: Start of 'ora.storage' on 'srv2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'srv2'

CRS-2676: Start of 'ora.crsd' on 'srv2' succeeded

[root@srv1 bin]# ps -ef |grep pmon
grid     11035     1  0 20:50 ?        00:00:00 asm_pmon_+ASM1
grid     11940     1  0 20:51 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   12166     1  0 20:51 ?        00:00:00 ora_pmon_rac1
root     19306 10230  0 20:59 pts/0    00:00:00 grep pmon
[root@srv1 bin]# ps -ef |grep lsnr
grid     11539     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr ASMIBNETLSNR_ASM -no_crs_notify -inherit
grid     11548     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
grid     11660     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid     11679     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid     11682     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     19425 11354  0 20:59 ?        00:00:00 /u01/app/12.2.0/grid/bin/lsnrctl status ASMIBNETLSNR_ASM
root     19438 10230  0 20:59 pts/0    00:00:00 grep lsnr

[oracle@srv2 bin]$ ps -ef |grep pmon

grid     12764     1  0 20:58 ?        00:00:00 asm_pmon_+ASM2

oracle   13686     1  0 20:59 ?        00:00:00 ora_pmon_rac2

oracle   14068  3329  0 20:59 pts/0    00:00:00 grep pmon

[oracle@srv2 bin]$ ps -ef |grep lsnr

grid     12022     1  0 20:57 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr ASMIBNETLSNR_ASM -no_crs_notify -inherit

grid     12309     1  0 20:58 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit

oracle   14132  3329  0 20:59 pts/0    00:00:00 grep lsnr

[root@srv1 bin]# crsctl status resource -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ASMIBNETLSNR_ASM.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.CRS.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.DATA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.FRA.dg

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  OFFLINE      srv2                     STABLE

ora.chad

               OFFLINE OFFLINE      srv1                     STABLE

               OFFLINE OFFLINE      srv2                     STABLE

ora.net1.network

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.ons

               ONLINE  ONLINE       srv1                     STABLE

               ONLINE  ONLINE       srv2                     STABLE

ora.proxy_advm

               OFFLINE OFFLINE      srv1                     STABLE

               OFFLINE OFFLINE      srv2                     STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       srv2                     STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       srv1                     STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       srv1                     STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.

                                                             168.10.1,STABLE

ora.asm

      1        ONLINE  ONLINE       srv1                     Started,STABLE

      2        ONLINE  ONLINE       srv2                     Started,STABLE

ora.cvu

      1        ONLINE  ONLINE       srv1                     STABLE

ora.mgmtdb

      1        ONLINE  ONLINE       srv1                     Open,STABLE

ora.qosmserver

      1        ONLINE  ONLINE       srv1                     STABLE

ora.rac.db

      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o

                                                             racle/product/12.2.0

                                                             /db_1,STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       srv2                     STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.srv1.vip

      1        ONLINE  ONLINE       srv1                     STABLE

ora.srv2.vip

      1        ONLINE  ONLINE       srv2                     STABLE

--------------------------------------------------------------------------------

[root@srv1 bin]#