Showing posts with label RAC Troubleshooting. Show all posts
Showing posts with label RAC Troubleshooting. Show all posts

Wednesday, 15 December 2021

OUI-10022:The target area /u01/oracle/gridbase/oraInventory cannot be used because it is in an invalid state.

 OUI-10022:The target area /u01/oracle/gridbase/oraInventory cannot be used because it is in an invalid state.

  • Scenario : 
We clean up the RAC machine on both VMbox and try to reinstall from scratch  ora inventory was different from old installation.



  • Check location on both node and correct the same . In our case inventory location was different so correct it or remove inventory location it will automatically add inventory_loc during installation.

[root@srv2 etc]# vi oraInst.loc 

inventory_loc=/u01/oracle/gridbase/oraInventory

inst_group=oinstall


[root@srv2 etc]# cat  /etc/oraInst.loc 

inventory_loc=/u01/oracle/gridbase/oraInventory

inst_group=oinstall

[root@srv2 etc]# 


  • We re run ./gridsetup.sh and found no issue set run smoothly.  
Posted by at No comments:
Labels: ,

Monday, 3 August 2020

Delete and Re-create MGMTDB database Grid 12R2 12.2.0.1.0 version


Scenario Preview:

We have 12c R2 12.2.0.1.0 two node RAC database. installed on OEL 6.7 operating system on virtual box. This is test database structure. All services started except MGMTDB database . After troubleshooting on logs we found cluseter ware services unable to connect MGTDB database and further found error on redo log file. Since we do not have RMAN backup to resolve this error so we decided to delete and re-create MGMTDB database.  You can follow metalink docs (Doc ID 1589394.1)

MGMTDB log Error :

 ORA-00742: Log read detects lost write in thread 1 sequence 10 block 44763
ORA-00312: online log 1 thread 1:
'+CRS/_MGMTDB/ONLINELOG/group_1.264.1043438559'
***********************************************************************
CRS alert log error:

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(CONNECT_TIMEOUT=60)(ADDRESS=(PROTOCOL=tcp)(HOST=srv-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=GIMR_DSCREP_10)(CID=(PROGRAM=ologgerd)(HOST=srv1.example.com)(USER=root)))(SECURITY=(ENCRYPTION_CLIENT=requested)))

  VERSION INFORMATION:
        TNS for Linux: Version 12.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
  Time: 02-AUG-2020 00:50:07
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

***********************************************************************
MGMTB Instance Error:

[grid@srv1 root]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@srv1 root]$ export ORACLE_SID=-MGMTDB
[grid@srv1 root]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sun Aug 2 00:18:29 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  8801008 bytes
Variable Size             436208912 bytes
Database Buffers          624951296 bytes
Redo Buffers                3780608 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread 1 sequence 10 block 44763
ORA-00312: online log 1 thread 1:
'+CRS/_MGMTDB/ONLINELOG/group_1.264.1043438559'


MGMTLSNR Listener status :

[root@srv1 trace]# lsnrctl status MGMTLSNR

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 02-AUG-2020 00:17:22

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MGMTLSNR)))
STATUS of the LISTENER
------------------------
Alias                     MGMTLSNR
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                01-AUG-2020 23:24:19
Uptime                    0 days 0 hr. 53 min. 3 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/srv1/mgmtlsnr/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MGMTLSNR)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.10.1)(PORT=1526)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.224.241)(PORT=1526)))
The listener supports no services
The command completed successfully


[grid@srv1 _mgmtdb]$ crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMIBNETLSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  OFFLINE      srv2                     STABLE
ora.chad
               ONLINE  OFFLINE      srv1                     STABLE
               ONLINE  OFFLINE      srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.proxy_advm
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.
                                                             168.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv2                     STABLE
ora.mgmtdb
      1        ONLINE  INTERMEDIATE srv1                     Mounted (Closed),Ope
                                                             n Initiated,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv2                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
[grid@srv1 _mgmtdb]$

Delelete MGMTDB steps:
Stop and disable ora.crf resource so it will not start automatically.

[root@srv1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
[root@srv1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@srv2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

Delete MGMTDB (grid user):

(since mgmtdb was running on another node srv2 and mgmtlsnr is running on srv1 so we got error below this is just example error)

Procedure to run mgmtdb on node srv1 (optional u can skip  if you do not face error below):

 [grid@srv1 bin]$ ./dbca -silent -deleteDatabase -sourceDB -MGMTDB
[FATAL] [DBT-10003] Delete operation for Oracle Grid Infrastructure Management Repository (GIMR) cannot be performed on the current node (srv1).
   CAUSE: Oracle GIMR is running on a remote node (srv2).
   ACTION: Invoke DBCA on the remote node (srv2) to delete Oracle GIMR.

srvctl stop mgmtdb (run on srv2 node with grid user)

srvctl start mgmtdb -n srv1 ( run on srv1 node with  grid user)

sqlplus  / as sysbda

startup mount;

Stop and disable ora.crf resource on each node so it will not start automatically on.

[root@srv1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
[root@srv1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@srv2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

 Delete command step:

[grid@srv1 bin]$ ./dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log" for further details.

[grid@srv1 bin]$ cat "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb.log"
[ 2020-08-02 19:55:46.748 IST ] The Database Configuration Assistant will delete the Oracle instance and datafiles for your database. All information in the database will be destroyed. Do you want to proceed?
[ 2020-08-02 19:55:46.914 IST ] Connecting to database
DBCA_PROGRESS : 4%
DBCA_PROGRESS : 9%
DBCA_PROGRESS : 14%
DBCA_PROGRESS : 19%
DBCA_PROGRESS : 23%
DBCA_PROGRESS : 28%
DBCA_PROGRESS : 47%
[ 2020-08-02 19:57:00.721 IST ] Updating network configuration files
DBCA_PROGRESS : 52%
[ 2020-08-02 19:57:00.790 IST ] Deleting instance and datafiles
DBCA_PROGRESS : 76%
DBCA_PROGRESS : 100%
[ 2020-08-02 19:57:47.860 IST ] Database deletion completed.


Delete MGMTDB at OS level Also:

[grid@srv1 bin]$
ASMCMD> cd _MGMTDB/
ASMCMD> ls
FD9AC0F7C36E4438E043B6A9E80A24D5/
FD9B43BF6A646F8CE043B6A9E80A2815/
ASMCMD> pwd
+CRS/_MGMTDB
ASMCMD> rm -rf *
ASMCMD>
[ 2020-08-02 19:57:47.860 IST ] Database deletion completed.

Create MGMTDB database command:

[grid@srv1 bin]$ ./dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
6% complete
7% complete
22% complete
36% complete
Creating and starting Oracle instance
37% complete
39% complete
43% complete
44% complete
45% complete
49% complete
53% complete
54% complete
55% complete
58% complete
Completing Database Creation
59% complete
62% complete
65% complete
68% complete
Executing Post Configuration Actions
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details.

In case PDB creation command in our case it is non-cdb rac 12R2 so we skip it:

 /u01/app/12.2.0/grid/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB \
-pdbName rac -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb \
-PDBMetadataFile /u01/app/12.2.0/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true

Start and enable ora.crf resource on each node so it will start automatically.

 [root@srv1 bin]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@srv1 bin]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'srv1'
CRS-2676: Start of 'ora.crf' on 'srv1' succeeded

[root@srv2 bin]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@srv2 bin]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'srv2'
CRS-2676: Start of 'ora.crf' on 'srv2' succeeded
[root@srv2 bin]#

Check the status of CRS resource:

grid@srv1 bin]$[root@srv1 bin]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMIBNETLSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  OFFLINE      srv2                     STABLE
ora.chad
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.proxy_advm
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.
                                                             168.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

We just Stop and start clusterware service again (optional):

[root@srv1 bin]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'srv2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv2'
CRS-2673: Attempting to stop 'ora.rac.db' on 'srv2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'srv2'
CRS-2673: Attempting to stop 'ora.crsd' on 'srv1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'srv1'
CRS-2673: Attempting to stop 'ora.rac.db' on 'srv1'
CRS-2677: Stop of 'ora.qosmserver' on 'srv2' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'srv1'
CRS-2677: Stop of 'ora.rac.db' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'srv2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'srv2'
CRS-2673: Attempting to stop 'ora.cvu' on 'srv2'
CRS-2673: Attempting to stop 'ora.srv2.vip' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'srv2'
CRS-2677: Stop of 'ora.DATA.dg' on 'srv2' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'srv2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv2'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'srv1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.srv2.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv2'
CRS-2677: Stop of 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'srv2' succeeded
CRS-2677: Stop of 'ora.rac.db' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'srv1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'srv1'
CRS-2677: Stop of 'ora.CRS.dg' on 'srv1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'srv1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'srv1'
CRS-2673: Attempting to stop 'ora.srv1.vip' on 'srv1'
CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv1'
CRS-2677: Stop of 'ora.scan1.vip' on 'srv1' succeeded
CRS-2677: Stop of 'ora.srv1.vip' on 'srv1' succeeded
CRS-2677: Stop of 'ora.ASMIBNETLSNR_ASM.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'srv1'
CRS-2677: Stop of 'ora.ons' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'srv1'
CRS-2677: Stop of 'ora.net1.network' on 'srv1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv1' has completed
CRS-2673: Attempting to stop 'ora.ons' on 'srv2'
CRS-2677: Stop of 'ora.ons' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'srv2'
CRS-2677: Stop of 'ora.net1.network' on 'srv2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'srv2' has completed
CRS-2677: Stop of 'ora.crsd' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'srv1'
CRS-2673: Attempting to stop 'ora.evmd' on 'srv1'
CRS-2673: Attempting to stop 'ora.storage' on 'srv1'
CRS-2677: Stop of 'ora.crsd' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'srv2'
CRS-2673: Attempting to stop 'ora.evmd' on 'srv2'
CRS-2673: Attempting to stop 'ora.storage' on 'srv2'
CRS-2677: Stop of 'ora.evmd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.storage' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv2'
CRS-2677: Stop of 'ora.evmd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.storage' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv1'
CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv1'
CRS-2677: Stop of 'ora.cssd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv2'
CRS-2677: Stop of 'ora.cssd' on 'srv2' succeeded

[root@srv1 bin]# crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv2'
CRS-2672: Attempting to start 'ora.evmd' on 'srv2'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv1'
CRS-2672: Attempting to start 'ora.evmd' on 'srv1'
CRS-2676: Start of 'ora.cssdmonitor' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'srv1'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv1'
CRS-2676: Start of 'ora.evmd' on 'srv1' succeeded
CRS-2676: Start of 'ora.diskmon' on 'srv1' succeeded
CRS-2676: Start of 'ora.evmd' on 'srv2' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'srv2'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv2'
CRS-2676: Start of 'ora.diskmon' on 'srv2' succeeded
CRS-2676: Start of 'ora.cssd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'srv1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv1'
CRS-2676: Start of 'ora.cssd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'srv2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2676: Start of 'ora.ctssd' on 'srv1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'srv2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv1'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv2'
CRS-2676: Start of 'ora.asm' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'srv2'
CRS-2676: Start of 'ora.asm' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'srv1'
CRS-2676: Start of 'ora.storage' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'srv1'
CRS-2676: Start of 'ora.crsd' on 'srv1' succeeded
CRS-2676: Start of 'ora.storage' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'srv2'
CRS-2676: Start of 'ora.crsd' on 'srv2' succeeded

[root@srv1 bin]# ps -ef |grep pmon
grid     11035     1  0 20:50 ?        00:00:00 asm_pmon_+ASM1
grid     11940     1  0 20:51 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   12166     1  0 20:51 ?        00:00:00 ora_pmon_rac1
root     19306 10230  0 20:59 pts/0    00:00:00 grep pmon
[root@srv1 bin]# ps -ef |grep lsnr
grid     11539     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr ASMIBNETLSNR_ASM -no_crs_notify -inherit
grid     11548     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
grid     11660     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid     11679     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid     11682     1  0 20:50 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     19425 11354  0 20:59 ?        00:00:00 /u01/app/12.2.0/grid/bin/lsnrctl status ASMIBNETLSNR_ASM
root     19438 10230  0 20:59 pts/0    00:00:00 grep lsnr

[oracle@srv2 bin]$ ps -ef |grep pmon
grid     12764     1  0 20:58 ?        00:00:00 asm_pmon_+ASM2
oracle   13686     1  0 20:59 ?        00:00:00 ora_pmon_rac2
oracle   14068  3329  0 20:59 pts/0    00:00:00 grep pmon
[oracle@srv2 bin]$ ps -ef |grep lsnr
grid     12022     1  0 20:57 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr ASMIBNETLSNR_ASM -no_crs_notify -inherit
grid     12309     1  0 20:58 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
oracle   14132  3329  0 20:59 pts/0    00:00:00 grep lsnr


[root@srv1 bin]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMIBNETLSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  OFFLINE      srv2                     STABLE
ora.chad
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.proxy_advm
               OFFLINE OFFLINE      srv1                     STABLE
               OFFLINE OFFLINE      srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.224.241 192.
                                                             168.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv1                     STABLE
ora.rac.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db_1,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
[root@srv1 bin]#

Posted by at No comments:
Labels:

Sunday, 26 July 2020

Troubleshooting Error: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line

Scenario Preview::

We face error while installing database software on 2 nodes . Permission of trace location got changed . We check | compare the permission on another node on same location and after changing group permission error got resolved.  

Error:

true 2 Additional information: 2
true 3 ([all diagnostic data retrieved from ADR])]
true 4 2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1422] return code: [ORA-49802] Oracle Base: [/u01/app/grid]  Product Type: [CRS]  Host Name: [srv1]  Instance ID: [crs]  User Name: [oracle]]
true 5 Additional information: 16893
true 6 1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure: 
true 7 Linux-x86_64 Error: 13: Permission denied
true 10 DIA-49801: actual permissions [rwxrwxr-x], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
true 11 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/u01/app/grid/diag/crs/srv1/crs/log]
true 12 DIA-48188: user missing read, write, or exec permission on specified directory
false 13 Oracle Clusterware infrastructure error in OLSNODES (OS PID 15159): CLSD/ADR initialization failed with return value -1
true 14 Additional information: 511







Changing the permission of grid directory  /u01/app/grid/diag/crs/srv1/crs

grid@srv1:/u01/app/grid/diag/crs/srv1/crs> ll
total 116
drwxrwxr-x 2 root root  4096 Jul 22 13:00 alert
drwxrwxr-x 2 root root  4096 Jul 22 09:37 cdump
drwxrwxr-x 2 root root  4096 Jul 22 09:37 incident
drwxrwxr-x 2 root root  4096 Jul 22 09:37 incpkg
drwxrwxr-x 2 root root  4096 Jul 22 09:37 lck
drwxrwxr-x 4 root root  4096 Jul 22 09:37 log
drwxrwxr-x 2 root root  4096 Jul 22 09:37 metadata
drwxrwxr-x 2 root root  4096 Jul 22 09:37 metadata_dgif
drwxrwxr-x 2 root root  4096 Jul 22 09:37 metadata_pv
drwxrwxr-x 2 root root  4096 Jul 22 09:37 stage
drwxrwxr-x 2 root root  4096 Jul 22 09:37 sweep
drwxrwxr-x 2 root root 69632 Jul 26 16:36 trace

srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall cdump
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall incident
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall incpkg
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall lck
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall log
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall metadata
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall metadata_dgif
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall metadata_pv
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall stage
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall sweep
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall trace
srv1:/u01/app/grid/diag/crs/srv1/crs # ll
total 116
drwxrwxr-x 2 root oinstall  4096 Jul 22 13:00 alert
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 cdump
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 incident
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 incpkg
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 lck
drwxrwxr-x 4 root oinstall  4096 Jul 22 09:37 log
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 metadata
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 metadata_dgif
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 metadata_pv
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 stage
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 sweep
drwxrwxr-x 2 root oinstall 69632 Jul 26 16:36 trace
srv1:/u01/app/grid/diag/crs/srv1/crs # ll


Issue resolved :

Posted by at No comments:
Labels:

ORA-27300: OS system dependent operation:fork failed with status:

Scenario Preview :

we have 2 node grid 12c . After upgrade unable to start instance on node1 so after check alertlog file we found some error below after troubleshooting  found doc (Doc ID 1546393.1) and increases value at OS level.

Operating System : SLES 12 SP4

 Error:

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process m000 died, see its trace file

alert_SMRPLM1.log

srv1:/u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace # tail -1000f alert_SMRPLM1.log
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Sun Jul 26 20:53:23 2020
Process startup failed, error stack:
Sun Jul 26 20:53:24 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process m000 died, see its trace file
Sun Jul 26 20:53:26 2020
Process startup failed, error stack:
Sun Jul 26 20:53:26 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Process m000 died, see its trace file
Sun Jul 26 20:53:46 2020
Instance reconfiguration: current system load 19 (hiload)
Sun Jul 26 20:53:52 2020
Reconfiguration started (old inc 2, new inc 4)
List of instances (total 2) :
 1 2
New instances (total 1) :
 2
My inst 1
Sun Jul 26 20:54:09 2020
 Global Resource Directory frozen
Starting background process RSMN
Sun Jul 26 20:54:09 2020
Process startup failed, error stack:
Sun Jul 26 20:54:09 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Sun Jul 26 20:54:10 2020
NOTE: ASMB connected to ASM instance +ASM1 osid: 6613 (Flex mode; client id 0x10001)
Sun Jul 26 20:54:10 2020
 Communication channels reestablished
Sun Jul 26 20:54:10 2020
NOTE: initiating MARK startup
Sun Jul 26 20:54:10 2020
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Process RSMN died, see its trace file
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Sun Jul 26 20:54:10 2020
 Post SMON to start 1st pass IR
 Submitted all GCS remote-cache requests
Sun Jul 26 20:54:11 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_ora_6432.trc:
ORA-00443: background process "RSMN" did not start
Sun Jul 26 20:54:11 2020
USER (ospid: 6432): terminating the instance due to error 450
Sun Jul 26 20:54:14 2020
Instance terminated by USER, pid = 6432

**********************************************************************


srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.185.74 192.1
                                                             68.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
ora.smrplm.pretafplm.svc
      1        ONLINE  ONLINE       srv2                     STABLE
ora.smrplm.pretafplm_preconnect.svc
      1        ONLINE  OFFLINE                               STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

 Solution :

Following  ORA-27300: OS System Dependent Operation:fork Failed With Status: 11 (Doc ID 1546393.1)

Increase the nporc value on file  /etc/security/limits.conf

oracle soft nproc 131072

grid soft nproc 131072

Also check  value on /etc/sysctl.conf file :

srv1:/home/oracle # sysctl -w kernel.pid_max=139264

kernel.pid_max = 139264

sysctl -p

oracle@srv1:~> sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Jul 26 21:04:50 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size            1140852856 bytes
Database Buffers          989855744 bytes
Redo Buffers               13848576 bytes
Database mounted.
Database opened.
SQL>

oracle@srv1:~> srvctl status database -d SMRPLM
Instance SMRPLM1 is running on node srv1
Instance SMRPLM2 is running on node srv2

Still in some case i have seen error occurred set value below at OS file  
 # cat /etc/systemd/system.conf set value DefaultTasksMax=65535

Posted by at No comments:
Labels:

Saturday, 25 July 2020

Troubleshooting - Grid Upgrade Error - [INS-20802] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.

Scenario Preview :

We recently doing upgrade activity on 12.1.0.2.0 to 12.2.0.1.0 grid 12c . We already executed /u01/app/12.2.0/grid/rootupgrade.sh on both the nodes and executed successfully.

But on 70% we face error [INS-20802] Creating Container Database for Oracle Grid Infrastructure Management Repository failed. After troubleshooting we found there is conflict in mgmtlsnr 1521 during mgmtdb installation due to this we are facing this issue, But when i check at my end the mgmtlsnr port was 1526 , Same issue i faced during 2 node RAC 12c installation on SLES 12 linux. but then mgmtlsnr was 1521. That scenario was different as at that time i have to install grid 12c on existing production databases on same  operating system SLES 12 SP4 . So i have changed the port of standalone DB . After that i have install grid 12c on same OS. It goes smoothly without error.

Previously i have done upgrade same scenario on OEL 6.7 opearting system grid upgrade done with out error. There is bug issue in SLES SUSE linux 12 SP4.

In this scenario mgmtlsnr was 1526 there shouldn't be no port conflict. But after so many troubleshooting and try i have follow some blogs and (Doc ID 2191588.1)  and installation completed successfully.
At the end cluster verification utility got failed this is the bug on SLES 12SP4 already mentioned in oracle metalink and we have to apply patch to resolve this issue. There is no affect on clusterware  performance in my production it is working perfectly fine. In OEL operating system i do not see any cluster verification utility error.  I have limitation to work in SUSE linux (:

Reference : 12c Grid Infrastructure Installation Fails While Creating Container Database: GIMR CRS-2800: Cannot start resource 'ora.MGMTLSNR' (Doc ID 2191588.1) 

 Error :

[INS-20802] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.

Cause - The plug-in failed in its perform method

Action - Refer to the logs or contact Oracle Support Services.  Log File Location
/u01/app/oraInventory/logs/GridSetupActions2020-07-24_06-04-56PM/gridSetupActions2020-07-24_06-04-56PM.log

Screen Shot Reference 


Grid Upgrade log error:

INFO:  [Jul 24, 2020 8:56:17 PM] Executing MGMTDBCDB
INFO:  [Jul 24, 2020 8:56:17 PM] Command /u01/app/12.2.0/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn.handleProcess() entered.
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn: getting configAssistantParmas.
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn: checking secretArguments.
INFO:  [Jul 24, 2020 8:56:17 PM] No arguments to pass to stdin
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn: starting read loop.
INFO:  [Jul 24, 2020 8:59:23 PM] Read: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 24, 2020 8:59:23 PM] Registering database with Oracle Grid Infrastructure
WARNING:  [Jul 24, 2020 8:59:23 PM] Skipping line: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 24, 2020 8:59:23 PM] End of argument passing to stdin
INFO:  [Jul 24, 2020 8:59:24 PM] Read: 5% complete
WARNING:  [Jul 24, 2020 8:59:24 PM] Skipping line: 5% complete
INFO:  [Jul 24, 2020 8:59:24 PM] Read: Copying database files
INFO:  [Jul 24, 2020 8:59:24 PM] Copying database files
WARNING:  [Jul 24, 2020 8:59:24 PM] Skipping line: Copying database files
INFO:  [Jul 24, 2020 8:59:24 PM] Read: 6% complete
WARNING:  [Jul 24, 2020 8:59:24 PM] Skipping line: 6% complete
INFO:  [Jul 24, 2020 8:59:52 PM] Read: 7% complete
WARNING:  [Jul 24, 2020 8:59:52 PM] Skipping line: 7% complete
INFO:  [Jul 24, 2020 9:00:49 PM] Read: 22% complete
WARNING:  [Jul 24, 2020 9:00:49 PM] Skipping line: 22% complete
INFO:  [Jul 24, 2020 9:00:49 PM] Read: 36% complete
WARNING:  [Jul 24, 2020 9:00:49 PM] Skipping line: 36% complete
INFO:  [Jul 24, 2020 9:00:49 PM] Read: Creating and starting Oracle instance
INFO:  [Jul 24, 2020 9:00:49 PM] Creating and starting Oracle instance
WARNING:  [Jul 24, 2020 9:00:49 PM] Skipping line: Creating and starting Oracle instance
INFO:  [Jul 24, 2020 9:02:10 PM] Read: 37% complete
WARNING:  [Jul 24, 2020 9:02:10 PM] Skipping line: 37% complete
INFO:  [Jul 24, 2020 9:02:41 PM] Read: 39% complete
WARNING:  [Jul 24, 2020 9:02:41 PM] Skipping line: 39% complete
INFO:  [Jul 24, 2020 9:06:59 PM] Read: 43% complete
WARNING:  [Jul 24, 2020 9:06:59 PM] Skipping line: 43% complete
INFO:  [Jul 24, 2020 9:07:36 PM] Read: 44% complete
WARNING:  [Jul 24, 2020 9:07:36 PM] Skipping line: 44% complete
INFO:  [Jul 24, 2020 9:08:24 PM] Read: 45% complete
WARNING:  [Jul 24, 2020 9:08:24 PM] Skipping line: 45% complete
INFO:  [Jul 24, 2020 9:09:44 PM] Read: 49% complete
WARNING:  [Jul 24, 2020 9:09:44 PM] Skipping line: 49% complete
INFO:  [Jul 24, 2020 9:09:45 PM] Read: 53% complete
WARNING:  [Jul 24, 2020 9:09:45 PM] Skipping line: 53% complete
INFO:  [Jul 24, 2020 9:11:09 PM] Read: 54% complete
WARNING:  [Jul 24, 2020 9:11:09 PM] Skipping line: 54% complete
INFO:  [Jul 24, 2020 9:11:14 PM] Read: 55% complete
WARNING:  [Jul 24, 2020 9:11:14 PM] Skipping line: 55% complete
INFO:  [Jul 24, 2020 9:11:50 PM] Read: 58% complete
WARNING:  [Jul 24, 2020 9:11:50 PM] Skipping line: 58% complete
INFO:  [Jul 24, 2020 9:11:50 PM] Read: Completing Database Creation
INFO:  [Jul 24, 2020 9:11:50 PM] Completing Database Creation
WARNING:  [Jul 24, 2020 9:11:50 PM] Skipping line: Completing Database Creation
INFO:  [Jul 24, 2020 9:11:51 PM] Read: 59% complete
WARNING:  [Jul 24, 2020 9:11:51 PM] Skipping line: 59% complete
INFO:  [Jul 24, 2020 9:12:08 PM] Read: 62% complete
WARNING:  [Jul 24, 2020 9:12:08 PM] Skipping line: 62% complete
INFO:  [Jul 24, 2020 9:21:13 PM] Read: 63% complete
WARNING:  [Jul 24, 2020 9:21:13 PM] Skipping line: 63% complete
INFO:  [Jul 24, 2020 9:22:39 PM] Read: DBCA Operation failed.
INFO:  [Jul 24, 2020 9:22:39 PM] DBCA Operation failed.
WARNING:  [Jul 24, 2020 9:22:39 PM] Skipping line: DBCA Operation failed.
INFO:  [Jul 24, 2020 9:22:39 PM] Read: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
INFO:  [Jul 24, 2020 9:22:39 PM] Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
WARNING:  [Jul 24, 2020 9:22:39 PM] Skipping line: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
INFO:  [Jul 24, 2020 9:22:42 PM] Completed Plugin named: Creating Container Database for Oracle Grid Infrastructure Management Repository
INFO:  [Jul 24, 2020 10:58:14 PM] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.
INFO:  [Jul 24, 2020 10:58:14 PM] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.

MGMTDB logs:

srv1:/u01/app/grid/cfgtoollogs/dbca/_mgmtdb # tail -1000f /u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb2.log
Registering database with Oracle Grid Infrastructure
DBCA_PROGRESS : 5%
 Copying database files
DBCA_PROGRESS : 6%
DBCA_PROGRESS : 7%
DBCA_PROGRESS : 22%
DBCA_PROGRESS : 36%
 Creating and starting Oracle instance
DBCA_PROGRESS : 37%
DBCA_PROGRESS : 39%
DBCA_PROGRESS : 43%
DBCA_PROGRESS : 44%
DBCA_PROGRESS : 45%
DBCA_PROGRESS : 49%
DBCA_PROGRESS : 53%
DBCA_PROGRESS : 54%
DBCA_PROGRESS : 55%
DBCA_PROGRESS : 58%
 Completing Database Creation
DBCA_PROGRESS : 59%
DBCA_PROGRESS : 62%
 ORA-01034: ORACLE not available

 ORA-01034: ORACLE not available

 ORA-01034: ORACLE not available

 DBCA_PROGRESS : DBCA Operation failed.
Solution :  (Doc ID 2191588.1) 

 srv1:~ # srvctl config MGMTLSNR

Name: MGMTLSNR
Type: Management Listener
Owner: grid
Home: <CRS home>
End points: TCP:1526
Management listener is enabled.
Management listener is individually enabled on nodes:
Management listener is individually disabled on nodes:
srv1:~ #

srv1:~ # srvctl stop mgmtlsnr -n srv1 -f

srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.185.74 192.1
                                                             68.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     STABLE
      2        ONLINE  ONLINE       srv2                     STABLE
ora.cvu
      1        ONLINE  ONLINE       srv2                     STABLE
ora.qosmserver
      1        ONLINE  OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
ora.smrplm.pretafplm.svc
      1        OFFLINE OFFLINE                               STABLE
ora.smrplm.pretafplm_preconnect.svc
      1        OFFLINE OFFLINE                               STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

srv1:~ # srvctl modify mgmtlsnr -endpoints "TCP:1531"

 srv1:~ # srvctl start MGMTLSNR

srv1:~ # srvctl config MGMTLSNR

Name: MGMTLSNR
Type: Management Listener
Owner: grid
Home: <CRS home>
End points: TCP:1531
Management listener is enabled.
Management listener is individually enabled on nodes:
Management listener is individually disabled on nodes:


Screen Shot Reference 

Upgrade Logs after error:

INFO:  [Jul 25, 2020 12:42:40 AM] Executing MGMTDBCDB
INFO:  [Jul 25, 2020 12:42:40 AM] Command /u01/app/12.2.0/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn.handleProcess() entered.
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn: getting configAssistantParmas.
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn: checking secretArguments.
INFO:  [Jul 25, 2020 12:42:40 AM] No arguments to pass to stdin
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn: starting read loop.
INFO:  [Jul 25, 2020 12:43:54 AM] Read: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 25, 2020 12:43:54 AM] Registering database with Oracle Grid Infrastructure
WARNING:  [Jul 25, 2020 12:43:54 AM] Skipping line: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 25, 2020 12:43:54 AM] End of argument passing to stdin
INFO:  [Jul 25, 2020 12:43:55 AM] Read: 5% complete
WARNING:  [Jul 25, 2020 12:43:55 AM] Skipping line: 5% complete
INFO:  [Jul 25, 2020 12:43:55 AM] Read: Copying database files
INFO:  [Jul 25, 2020 12:43:55 AM] Copying database files
WARNING:  [Jul 25, 2020 12:43:55 AM] Skipping line: Copying database files
INFO:  [Jul 25, 2020 12:43:56 AM] Read: 6% complete
WARNING:  [Jul 25, 2020 12:43:56 AM] Skipping line: 6% complete
INFO:  [Jul 25, 2020 12:44:14 AM] Read: 7% complete
WARNING:  [Jul 25, 2020 12:44:14 AM] Skipping line: 7% complete
INFO:  [Jul 25, 2020 12:45:00 AM] Read: 22% complete
WARNING:  [Jul 25, 2020 12:45:00 AM] Skipping line: 22% complete
INFO:  [Jul 25, 2020 12:45:00 AM] Read: 36% complete
WARNING:  [Jul 25, 2020 12:45:00 AM] Skipping line: 36% complete
INFO:  [Jul 25, 2020 12:45:00 AM] Read: Creating and starting Oracle instance
INFO:  [Jul 25, 2020 12:45:00 AM] Creating and starting Oracle instance
WARNING:  [Jul 25, 2020 12:45:00 AM] Skipping line: Creating and starting Oracle instance
INFO:  [Jul 25, 2020 12:45:25 AM] Read: 37% complete
WARNING:  [Jul 25, 2020 12:45:25 AM] Skipping line: 37% complete
INFO:  [Jul 25, 2020 12:45:56 AM] Read: 39% complete
WARNING:  [Jul 25, 2020 12:45:56 AM] Skipping line: 39% complete
INFO:  [Jul 25, 2020 12:49:40 AM] Read: 43% complete
WARNING:  [Jul 25, 2020 12:49:40 AM] Skipping line: 43% complete
INFO:  [Jul 25, 2020 12:50:09 AM] Read: 44% complete
WARNING:  [Jul 25, 2020 12:50:09 AM] Skipping line: 44% complete
INFO:  [Jul 25, 2020 12:50:40 AM] Read: 45% complete
WARNING:  [Jul 25, 2020 12:50:40 AM] Skipping line: 45% complete
INFO:  [Jul 25, 2020 12:51:54 AM] Read: 49% complete
WARNING:  [Jul 25, 2020 12:51:54 AM] Skipping line: 49% complete
INFO:  [Jul 25, 2020 12:51:55 AM] Read: 53% complete
WARNING:  [Jul 25, 2020 12:51:55 AM] Skipping line: 53% complete
INFO:  [Jul 25, 2020 12:52:57 AM] Read: 54% complete
WARNING:  [Jul 25, 2020 12:52:57 AM] Skipping line: 54% complete
INFO:  [Jul 25, 2020 12:53:01 AM] Read: 55% complete
WARNING:  [Jul 25, 2020 12:53:01 AM] Skipping line: 55% complete
INFO:  [Jul 25, 2020 12:53:28 AM] Read: 58% complete
WARNING:  [Jul 25, 2020 12:53:28 AM] Skipping line: 58% complete
INFO:  [Jul 25, 2020 12:53:28 AM] Read: Completing Database Creation
INFO:  [Jul 25, 2020 12:53:28 AM] Completing Database Creation
WARNING:  [Jul 25, 2020 12:53:28 AM] Skipping line: Completing Database Creation
INFO:  [Jul 25, 2020 12:53:28 AM] Read: 59% complete
WARNING:  [Jul 25, 2020 12:53:28 AM] Skipping line: 59% complete
INFO:  [Jul 25, 2020 12:53:42 AM] Read: 62% complete
WARNING:  [Jul 25, 2020 12:53:42 AM] Skipping line: 62% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: 65% complete
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: 65% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: 68% complete
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: 68% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: Executing Post Configuration Actions
INFO:  [Jul 25, 2020 12:56:33 AM] Executing Post Configuration Actions
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: Executing Post Configuration Actions
INFO:  [Jul 25, 2020 12:56:33 AM] Read: 100% complete
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: 100% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
INFO:  [Jul 25, 2020 12:56:33 AM] Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
INFO:  [Jul 25, 2020 12:56:34 AM] Completed Plugin named: Creating Container Database for Oracle Grid Infrastructure Management Repository
INFO:  [Jul 25, 2020 12:56:34 AM] Completed 'Creating Container Database for Oracle Grid Infrastructure Management Repository'
INFO:  [Jul 25, 2020 12:56:34 AM] Completed 'Creating Container Database for Oracle Grid Infrastructure Management Repository'
INFO:  [Jul 25, 2020 12:56:34 AM] Started Plugin named: GIMR Configuration Assistant
INFO:  [Jul 25, 2020 12:56:34 AM] Found associated job
INFO:  [Jul 25, 2020 12:56:34 AM] Starting 'GIMR Configuration Assistant'
INFO:  [Jul 25, 2020 12:56:34 AM] Starting 'GIMR Configuration Assistant'
INFO:  [Jul 25, 2020 12:56:34 AM] Executing MGMTCA
INFO:  [Jul 25, 2020 12:56:34 AM] Command /u01/app/12.2.0/grid/bin/mgmtca -local
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn.handleProcess() entered.
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn: getting configAssistantParmas.
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn: checking secretArguments.
INFO:  [Jul 25, 2020 12:56:34 AM] No arguments to pass to stdin
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn: starting read loop.


Clusterware Commands:

srv2:~ # crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]
srv2:~ # crsctl query crs softwareversion
Oracle Clusterware version on node [srv2] is [12.2.0.1.0]
srv2:~ #

srv2:~ # crsctl check cluster -all
**************************************************************
srv1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
srv2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
srv2:~ #
srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.185.74 192.1
                                                             68.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
ora.smrplm.pretafplm.svc
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.pretafplm_preconnect.svc
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
srv1:~ #








Posted by at No comments:
Labels:
Subscribe to: Posts (Atom)