Sunday 26 July 2020

Troubleshooting Error: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line

Scenario Preview::

We face error while installing database software on 2 nodes . Permission of trace location got changed . We check | compare the permission on another node on same location and after changing group permission error got resolved.  

Error:

true 2 Additional information: 2
true 3 ([all diagnostic data retrieved from ADR])]
true 4 2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1422] return code: [ORA-49802] Oracle Base: [/u01/app/grid]  Product Type: [CRS]  Host Name: [srv1]  Instance ID: [crs]  User Name: [oracle]]
true 5 Additional information: 16893
true 6 1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure: 
true 7 Linux-x86_64 Error: 13: Permission denied
true 10 DIA-49801: actual permissions [rwxrwxr-x], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
true 11 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/u01/app/grid/diag/crs/srv1/crs/log]
true 12 DIA-48188: user missing read, write, or exec permission on specified directory
false 13 Oracle Clusterware infrastructure error in OLSNODES (OS PID 15159): CLSD/ADR initialization failed with return value -1
true 14 Additional information: 511







Changing the permission of grid directory  /u01/app/grid/diag/crs/srv1/crs

grid@srv1:/u01/app/grid/diag/crs/srv1/crs> ll
total 116
drwxrwxr-x 2 root root  4096 Jul 22 13:00 alert
drwxrwxr-x 2 root root  4096 Jul 22 09:37 cdump
drwxrwxr-x 2 root root  4096 Jul 22 09:37 incident
drwxrwxr-x 2 root root  4096 Jul 22 09:37 incpkg
drwxrwxr-x 2 root root  4096 Jul 22 09:37 lck
drwxrwxr-x 4 root root  4096 Jul 22 09:37 log
drwxrwxr-x 2 root root  4096 Jul 22 09:37 metadata
drwxrwxr-x 2 root root  4096 Jul 22 09:37 metadata_dgif
drwxrwxr-x 2 root root  4096 Jul 22 09:37 metadata_pv
drwxrwxr-x 2 root root  4096 Jul 22 09:37 stage
drwxrwxr-x 2 root root  4096 Jul 22 09:37 sweep
drwxrwxr-x 2 root root 69632 Jul 26 16:36 trace

srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall cdump
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall incident
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall incpkg
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall lck
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall log
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall metadata
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall metadata_dgif
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall metadata_pv
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall stage
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall sweep
srv1:/u01/app/grid/diag/crs/srv1/crs # chown -R root:oinstall trace
srv1:/u01/app/grid/diag/crs/srv1/crs # ll
total 116
drwxrwxr-x 2 root oinstall  4096 Jul 22 13:00 alert
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 cdump
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 incident
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 incpkg
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 lck
drwxrwxr-x 4 root oinstall  4096 Jul 22 09:37 log
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 metadata
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 metadata_dgif
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 metadata_pv
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 stage
drwxrwxr-x 2 root oinstall  4096 Jul 22 09:37 sweep
drwxrwxr-x 2 root oinstall 69632 Jul 26 16:36 trace
srv1:/u01/app/grid/diag/crs/srv1/crs # ll


Issue resolved :

ORA-27300: OS system dependent operation:fork failed with status:

Scenario Preview :

we have 2 node grid 12c . After upgrade unable to start instance on node1 so after check alertlog file we found some error below after troubleshooting  found doc (Doc ID 1546393.1) and increases value at OS level.

Operating System : SLES 12 SP4

Error:

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process m000 died, see its trace file

alert_SMRPLM1.log

srv1:/u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace # tail -1000f alert_SMRPLM1.log
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Sun Jul 26 20:53:23 2020
Process startup failed, error stack:
Sun Jul 26 20:53:24 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process m000 died, see its trace file
Sun Jul 26 20:53:26 2020
Process startup failed, error stack:
Sun Jul 26 20:53:26 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Process m000 died, see its trace file
Sun Jul 26 20:53:46 2020
Instance reconfiguration: current system load 19 (hiload)
Sun Jul 26 20:53:52 2020
Reconfiguration started (old inc 2, new inc 4)
List of instances (total 2) :
 1 2
New instances (total 1) :
 2
My inst 1
Sun Jul 26 20:54:09 2020
 Global Resource Directory frozen
Starting background process RSMN
Sun Jul 26 20:54:09 2020
Process startup failed, error stack:
Sun Jul 26 20:54:09 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Sun Jul 26 20:54:10 2020
NOTE: ASMB connected to ASM instance +ASM1 osid: 6613 (Flex mode; client id 0x10001)
Sun Jul 26 20:54:10 2020
 Communication channels reestablished
Sun Jul 26 20:54:10 2020
NOTE: initiating MARK startup
Sun Jul 26 20:54:10 2020
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Process RSMN died, see its trace file
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Sun Jul 26 20:54:10 2020
 Post SMON to start 1st pass IR
 Submitted all GCS remote-cache requests
Sun Jul 26 20:54:11 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_ora_6432.trc:
ORA-00443: background process "RSMN" did not start
Sun Jul 26 20:54:11 2020
USER (ospid: 6432): terminating the instance due to error 450
Sun Jul 26 20:54:14 2020
Instance terminated by USER, pid = 6432

**********************************************************************


srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.185.74 192.1
                                                             68.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
ora.smrplm.pretafplm.svc
      1        ONLINE  ONLINE       srv2                     STABLE
ora.smrplm.pretafplm_preconnect.svc
      1        ONLINE  OFFLINE                               STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

Solution :

Following  ORA-27300: OS System Dependent Operation:fork Failed With Status: 11 (Doc ID 1546393.1)



Increase the nporc value on file  /etc/security/limits.conf

oracle soft nproc 131072

grid soft nproc 131072

Also check  value on /etc/sysctl.conf file :

srv1:/home/oracle # sysctl -w kernel.pid_max=139264

kernel.pid_max = 139264

sysctl -p

oracle@srv1:~> sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Jul 26 21:04:50 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size            1140852856 bytes
Database Buffers          989855744 bytes
Redo Buffers               13848576 bytes
Database mounted.
Database opened.
SQL>

oracle@srv1:~> srvctl status database -d SMRPLM
Instance SMRPLM1 is running on node srv1
Instance SMRPLM2 is running on node srv2

Still in some case i have seen error occurred set value below at OS file  
 # cat /etc/systemd/system.conf set value DefaultTasksMax=65535

Saturday 25 July 2020

Troubleshooting - Grid Upgrade Error - [INS-20802] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.

Scenario Preview :

We recently doing upgrade activity on 12.1.0.2.0 to 12.2.0.1.0 grid 12c . We already executed /u01/app/12.2.0/grid/rootupgrade.sh on both the nodes and executed successfully.

But on 70% we face error [INS-20802] Creating Container Database for Oracle Grid Infrastructure Management Repository failed. After troubleshooting we found there is conflict in mgmtlsnr 1521 during mgmtdb installation due to this we are facing this issue, But when i check at my end the mgmtlsnr port was 1526 , Same issue i faced during 2 node RAC 12c installation on SLES 12 linux. but then mgmtlsnr was 1521. That scenario was different as at that time i have to install grid 12c on existing production databases on same  operating system SLES 12 SP4 . So i have changed the port of standalone DB . After that i have install grid 12c on same OS. It goes smoothly without error.

Previously i have done upgrade same scenario on OEL 6.7 opearting system grid upgrade done with out error. There is bug issue in SLES SUSE linux 12 SP4.

In this scenario mgmtlsnr was 1526 there shouldn't be no port conflict. But after so many troubleshooting and try i have follow some blogs and (Doc ID 2191588.1)  and installation completed successfully.
At the end cluster verification utility got failed this is the bug on SLES 12SP4 already mentioned in oracle metalink and we have to apply patch to resolve this issue. There is no affect on clusterware  performance in my production it is working perfectly fine. In OEL operating system i do not see any cluster verification utility error.  I have limitation to work in SUSE linux (:

Reference : 12c Grid Infrastructure Installation Fails While Creating Container Database: GIMR CRS-2800: Cannot start resource 'ora.MGMTLSNR' (Doc ID 2191588.1) 

Error :

[INS-20802] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.

Cause - The plug-in failed in its perform method

Action - Refer to the logs or contact Oracle Support Services.  Log File Location
/u01/app/oraInventory/logs/GridSetupActions2020-07-24_06-04-56PM/gridSetupActions2020-07-24_06-04-56PM.log

Screen Shot Reference 


Grid Upgrade log error:

INFO:  [Jul 24, 2020 8:56:17 PM] Executing MGMTDBCDB
INFO:  [Jul 24, 2020 8:56:17 PM] Command /u01/app/12.2.0/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn.handleProcess() entered.
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn: getting configAssistantParmas.
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn: checking secretArguments.
INFO:  [Jul 24, 2020 8:56:17 PM] No arguments to pass to stdin
INFO:  [Jul 24, 2020 8:56:17 PM] ... GenericInternalPlugIn: starting read loop.
INFO:  [Jul 24, 2020 8:59:23 PM] Read: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 24, 2020 8:59:23 PM] Registering database with Oracle Grid Infrastructure
WARNING:  [Jul 24, 2020 8:59:23 PM] Skipping line: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 24, 2020 8:59:23 PM] End of argument passing to stdin
INFO:  [Jul 24, 2020 8:59:24 PM] Read: 5% complete
WARNING:  [Jul 24, 2020 8:59:24 PM] Skipping line: 5% complete
INFO:  [Jul 24, 2020 8:59:24 PM] Read: Copying database files
INFO:  [Jul 24, 2020 8:59:24 PM] Copying database files
WARNING:  [Jul 24, 2020 8:59:24 PM] Skipping line: Copying database files
INFO:  [Jul 24, 2020 8:59:24 PM] Read: 6% complete
WARNING:  [Jul 24, 2020 8:59:24 PM] Skipping line: 6% complete
INFO:  [Jul 24, 2020 8:59:52 PM] Read: 7% complete
WARNING:  [Jul 24, 2020 8:59:52 PM] Skipping line: 7% complete
INFO:  [Jul 24, 2020 9:00:49 PM] Read: 22% complete
WARNING:  [Jul 24, 2020 9:00:49 PM] Skipping line: 22% complete
INFO:  [Jul 24, 2020 9:00:49 PM] Read: 36% complete
WARNING:  [Jul 24, 2020 9:00:49 PM] Skipping line: 36% complete
INFO:  [Jul 24, 2020 9:00:49 PM] Read: Creating and starting Oracle instance
INFO:  [Jul 24, 2020 9:00:49 PM] Creating and starting Oracle instance
WARNING:  [Jul 24, 2020 9:00:49 PM] Skipping line: Creating and starting Oracle instance
INFO:  [Jul 24, 2020 9:02:10 PM] Read: 37% complete
WARNING:  [Jul 24, 2020 9:02:10 PM] Skipping line: 37% complete
INFO:  [Jul 24, 2020 9:02:41 PM] Read: 39% complete
WARNING:  [Jul 24, 2020 9:02:41 PM] Skipping line: 39% complete
INFO:  [Jul 24, 2020 9:06:59 PM] Read: 43% complete
WARNING:  [Jul 24, 2020 9:06:59 PM] Skipping line: 43% complete
INFO:  [Jul 24, 2020 9:07:36 PM] Read: 44% complete
WARNING:  [Jul 24, 2020 9:07:36 PM] Skipping line: 44% complete
INFO:  [Jul 24, 2020 9:08:24 PM] Read: 45% complete
WARNING:  [Jul 24, 2020 9:08:24 PM] Skipping line: 45% complete
INFO:  [Jul 24, 2020 9:09:44 PM] Read: 49% complete
WARNING:  [Jul 24, 2020 9:09:44 PM] Skipping line: 49% complete
INFO:  [Jul 24, 2020 9:09:45 PM] Read: 53% complete
WARNING:  [Jul 24, 2020 9:09:45 PM] Skipping line: 53% complete
INFO:  [Jul 24, 2020 9:11:09 PM] Read: 54% complete
WARNING:  [Jul 24, 2020 9:11:09 PM] Skipping line: 54% complete
INFO:  [Jul 24, 2020 9:11:14 PM] Read: 55% complete
WARNING:  [Jul 24, 2020 9:11:14 PM] Skipping line: 55% complete
INFO:  [Jul 24, 2020 9:11:50 PM] Read: 58% complete
WARNING:  [Jul 24, 2020 9:11:50 PM] Skipping line: 58% complete
INFO:  [Jul 24, 2020 9:11:50 PM] Read: Completing Database Creation
INFO:  [Jul 24, 2020 9:11:50 PM] Completing Database Creation
WARNING:  [Jul 24, 2020 9:11:50 PM] Skipping line: Completing Database Creation
INFO:  [Jul 24, 2020 9:11:51 PM] Read: 59% complete
WARNING:  [Jul 24, 2020 9:11:51 PM] Skipping line: 59% complete
INFO:  [Jul 24, 2020 9:12:08 PM] Read: 62% complete
WARNING:  [Jul 24, 2020 9:12:08 PM] Skipping line: 62% complete
INFO:  [Jul 24, 2020 9:21:13 PM] Read: 63% complete
WARNING:  [Jul 24, 2020 9:21:13 PM] Skipping line: 63% complete
INFO:  [Jul 24, 2020 9:22:39 PM] Read: DBCA Operation failed.
INFO:  [Jul 24, 2020 9:22:39 PM] DBCA Operation failed.
WARNING:  [Jul 24, 2020 9:22:39 PM] Skipping line: DBCA Operation failed.
INFO:  [Jul 24, 2020 9:22:39 PM] Read: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
INFO:  [Jul 24, 2020 9:22:39 PM] Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
WARNING:  [Jul 24, 2020 9:22:39 PM] Skipping line: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
INFO:  [Jul 24, 2020 9:22:42 PM] Completed Plugin named: Creating Container Database for Oracle Grid Infrastructure Management Repository
INFO:  [Jul 24, 2020 10:58:14 PM] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.
INFO:  [Jul 24, 2020 10:58:14 PM] Creating Container Database for Oracle Grid Infrastructure Management Repository failed.

MGMTDB logs:

srv1:/u01/app/grid/cfgtoollogs/dbca/_mgmtdb # tail -1000f /u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb2.log
Registering database with Oracle Grid Infrastructure
DBCA_PROGRESS : 5%
 Copying database files
DBCA_PROGRESS : 6%
DBCA_PROGRESS : 7%
DBCA_PROGRESS : 22%
DBCA_PROGRESS : 36%
 Creating and starting Oracle instance
DBCA_PROGRESS : 37%
DBCA_PROGRESS : 39%
DBCA_PROGRESS : 43%
DBCA_PROGRESS : 44%
DBCA_PROGRESS : 45%
DBCA_PROGRESS : 49%
DBCA_PROGRESS : 53%
DBCA_PROGRESS : 54%
DBCA_PROGRESS : 55%
DBCA_PROGRESS : 58%
 Completing Database Creation
DBCA_PROGRESS : 59%
DBCA_PROGRESS : 62%
 ORA-01034: ORACLE not available

 ORA-01034: ORACLE not available

 ORA-01034: ORACLE not available

 DBCA_PROGRESS : DBCA Operation failed.



Solution :  (Doc ID 2191588.1) 

srv1:~ # srvctl config MGMTLSNR

Name: MGMTLSNR
Type: Management Listener
Owner: grid
Home: <CRS home>
End points: TCP:1526
Management listener is enabled.
Management listener is individually enabled on nodes:
Management listener is individually disabled on nodes:
srv1:~ #

srv1:~ # srvctl stop mgmtlsnr -n srv1 -f

srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.185.74 192.1
                                                             68.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     STABLE
      2        ONLINE  ONLINE       srv2                     STABLE
ora.cvu
      1        ONLINE  ONLINE       srv2                     STABLE
ora.qosmserver
      1        ONLINE  OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
ora.smrplm.pretafplm.svc
      1        OFFLINE OFFLINE                               STABLE
ora.smrplm.pretafplm_preconnect.svc
      1        OFFLINE OFFLINE                               STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------

srv1:~ # srvctl modify mgmtlsnr -endpoints "TCP:1531"

srv1:~ # srvctl start MGMTLSNR

srv1:~ # srvctl config MGMTLSNR

Name: MGMTLSNR
Type: Management Listener
Owner: grid
Home: <CRS home>
End points: TCP:1531
Management listener is enabled.
Management listener is individually enabled on nodes:
Management listener is individually disabled on nodes:


Screen Shot Reference 

Upgrade Logs after error:

INFO:  [Jul 25, 2020 12:42:40 AM] Executing MGMTDBCDB
INFO:  [Jul 25, 2020 12:42:40 AM] Command /u01/app/12.2.0/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName CRS -datafileJarLocation /u01/app/12.2.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn.handleProcess() entered.
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn: getting configAssistantParmas.
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn: checking secretArguments.
INFO:  [Jul 25, 2020 12:42:40 AM] No arguments to pass to stdin
INFO:  [Jul 25, 2020 12:42:40 AM] ... GenericInternalPlugIn: starting read loop.
INFO:  [Jul 25, 2020 12:43:54 AM] Read: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 25, 2020 12:43:54 AM] Registering database with Oracle Grid Infrastructure
WARNING:  [Jul 25, 2020 12:43:54 AM] Skipping line: Registering database with Oracle Grid Infrastructure
INFO:  [Jul 25, 2020 12:43:54 AM] End of argument passing to stdin
INFO:  [Jul 25, 2020 12:43:55 AM] Read: 5% complete
WARNING:  [Jul 25, 2020 12:43:55 AM] Skipping line: 5% complete
INFO:  [Jul 25, 2020 12:43:55 AM] Read: Copying database files
INFO:  [Jul 25, 2020 12:43:55 AM] Copying database files
WARNING:  [Jul 25, 2020 12:43:55 AM] Skipping line: Copying database files
INFO:  [Jul 25, 2020 12:43:56 AM] Read: 6% complete
WARNING:  [Jul 25, 2020 12:43:56 AM] Skipping line: 6% complete
INFO:  [Jul 25, 2020 12:44:14 AM] Read: 7% complete
WARNING:  [Jul 25, 2020 12:44:14 AM] Skipping line: 7% complete
INFO:  [Jul 25, 2020 12:45:00 AM] Read: 22% complete
WARNING:  [Jul 25, 2020 12:45:00 AM] Skipping line: 22% complete
INFO:  [Jul 25, 2020 12:45:00 AM] Read: 36% complete
WARNING:  [Jul 25, 2020 12:45:00 AM] Skipping line: 36% complete
INFO:  [Jul 25, 2020 12:45:00 AM] Read: Creating and starting Oracle instance
INFO:  [Jul 25, 2020 12:45:00 AM] Creating and starting Oracle instance
WARNING:  [Jul 25, 2020 12:45:00 AM] Skipping line: Creating and starting Oracle instance
INFO:  [Jul 25, 2020 12:45:25 AM] Read: 37% complete
WARNING:  [Jul 25, 2020 12:45:25 AM] Skipping line: 37% complete
INFO:  [Jul 25, 2020 12:45:56 AM] Read: 39% complete
WARNING:  [Jul 25, 2020 12:45:56 AM] Skipping line: 39% complete
INFO:  [Jul 25, 2020 12:49:40 AM] Read: 43% complete
WARNING:  [Jul 25, 2020 12:49:40 AM] Skipping line: 43% complete
INFO:  [Jul 25, 2020 12:50:09 AM] Read: 44% complete
WARNING:  [Jul 25, 2020 12:50:09 AM] Skipping line: 44% complete
INFO:  [Jul 25, 2020 12:50:40 AM] Read: 45% complete
WARNING:  [Jul 25, 2020 12:50:40 AM] Skipping line: 45% complete
INFO:  [Jul 25, 2020 12:51:54 AM] Read: 49% complete
WARNING:  [Jul 25, 2020 12:51:54 AM] Skipping line: 49% complete
INFO:  [Jul 25, 2020 12:51:55 AM] Read: 53% complete
WARNING:  [Jul 25, 2020 12:51:55 AM] Skipping line: 53% complete
INFO:  [Jul 25, 2020 12:52:57 AM] Read: 54% complete
WARNING:  [Jul 25, 2020 12:52:57 AM] Skipping line: 54% complete
INFO:  [Jul 25, 2020 12:53:01 AM] Read: 55% complete
WARNING:  [Jul 25, 2020 12:53:01 AM] Skipping line: 55% complete
INFO:  [Jul 25, 2020 12:53:28 AM] Read: 58% complete
WARNING:  [Jul 25, 2020 12:53:28 AM] Skipping line: 58% complete
INFO:  [Jul 25, 2020 12:53:28 AM] Read: Completing Database Creation
INFO:  [Jul 25, 2020 12:53:28 AM] Completing Database Creation
WARNING:  [Jul 25, 2020 12:53:28 AM] Skipping line: Completing Database Creation
INFO:  [Jul 25, 2020 12:53:28 AM] Read: 59% complete
WARNING:  [Jul 25, 2020 12:53:28 AM] Skipping line: 59% complete
INFO:  [Jul 25, 2020 12:53:42 AM] Read: 62% complete
WARNING:  [Jul 25, 2020 12:53:42 AM] Skipping line: 62% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: 65% complete
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: 65% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: 68% complete
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: 68% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: Executing Post Configuration Actions
INFO:  [Jul 25, 2020 12:56:33 AM] Executing Post Configuration Actions
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: Executing Post Configuration Actions
INFO:  [Jul 25, 2020 12:56:33 AM] Read: 100% complete
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: 100% complete
INFO:  [Jul 25, 2020 12:56:33 AM] Read: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
INFO:  [Jul 25, 2020 12:56:33 AM] Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
WARNING:  [Jul 25, 2020 12:56:33 AM] Skipping line: Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
INFO:  [Jul 25, 2020 12:56:34 AM] Completed Plugin named: Creating Container Database for Oracle Grid Infrastructure Management Repository
INFO:  [Jul 25, 2020 12:56:34 AM] Completed 'Creating Container Database for Oracle Grid Infrastructure Management Repository'
INFO:  [Jul 25, 2020 12:56:34 AM] Completed 'Creating Container Database for Oracle Grid Infrastructure Management Repository'
INFO:  [Jul 25, 2020 12:56:34 AM] Started Plugin named: GIMR Configuration Assistant
INFO:  [Jul 25, 2020 12:56:34 AM] Found associated job
INFO:  [Jul 25, 2020 12:56:34 AM] Starting 'GIMR Configuration Assistant'
INFO:  [Jul 25, 2020 12:56:34 AM] Starting 'GIMR Configuration Assistant'
INFO:  [Jul 25, 2020 12:56:34 AM] Executing MGMTCA
INFO:  [Jul 25, 2020 12:56:34 AM] Command /u01/app/12.2.0/grid/bin/mgmtca -local
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn.handleProcess() entered.
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn: getting configAssistantParmas.
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn: checking secretArguments.
INFO:  [Jul 25, 2020 12:56:34 AM] No arguments to pass to stdin
INFO:  [Jul 25, 2020 12:56:34 AM] ... GenericInternalPlugIn: starting read loop.


Clusterware Commands:

srv2:~ # crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]
srv2:~ # crsctl query crs softwareversion
Oracle Clusterware version on node [srv2] is [12.2.0.1.0]
srv2:~ #

srv2:~ # crsctl check cluster -all
**************************************************************
srv1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
srv2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
srv2:~ #
srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.185.74 192.1
                                                             68.10.1,STABLE
ora.asm
      1        ONLINE  ONLINE       srv1                     Started,STABLE
      2        ONLINE  ONLINE       srv2                     Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  ONLINE       srv1                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       srv2                     Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             /dbhome_1,STABLE
ora.smrplm.pretafplm.svc
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.pretafplm_preconnect.svc
      1        ONLINE  ONLINE       srv2                     STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
srv1:~ #








Wednesday 22 July 2020

Multitenant CDB Database creation using template on RAC 12c

Scenario Preview:

Creation of Multitenant database on 12.1.0.2.0 using team centre teamplate  on virtual box test machine.
We have already installed 2 node grid 12c and database software already installed.


Over view details of template files used for dbca creation  . Unable to share files.


Two files mainly Teamcenter_Oracle.dbt and Teamcenter_Oracle_multitenant.dbt used for template rest files are used to create user and privilages.
If we wants to create non CDB database using template use Teamcenter_Oracle.dbt and for CDB use 
Teamcenter_Oracle_multitenant.dbt

Details below:

Teamcenter_Oracle.dbt :

<DatabaseTemplate name="tc" description="" version="12.1.0.1.0">
   <CommonAttributes>
      <option name="ISEARCH" value="false"/>
      <option name="OMS" value="false"/>
      <option name="JSERVER" value="false"/>
      <option name="SPATIAL" value="false"/>
      <option name="ODM" value="false">
         <tablespace id="SYSAUX"/>
      </option>
      <option name="IMEDIA" value="false"/>
      <option name="XDB_PROTOCOLS" value="false">
         <tablespace id="SYSAUX"/>
      </option>
      <option name="ORACLE_TEXT" value="false">
         <tablespace id="SYSAUX"/>
      </option>
      <option name="SAMPLE_SCHEMA" value="false"/>
      <option name="CWMLITE" value="false">
         <tablespace id="SYSAUX"/>
      </option>
      <option name="EM_REPOSITORY" value="true">
         <tablespace id="SYSAUX"/>
      </option>
   </CommonAttributes>
   <Variables/>
   <CustomScripts Execute="true">
      <CustomScript script="{ORACLE_HOME}/assistants/dbca/templates/tc_create_user_ilog.sql"/>
   </CustomScripts>
   <InitParamAttributes>
      <InitParams>
         <initParam name="control_files" value="(&quot;{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control01.ctl&quot;, &quot;{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control02.ctl&quot;, &quot;{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control03.ctl&quot;)"/>
         <initParam name="db_domain" value=""/>
         <initParam name="db_file_multiblock_read_count" value="32"/>
         <initParam name="db_name" value="tc"/>
         <initParam name="instance_name" value="tc"/>
         <initParam name="db_recovery_file_dest" value="{ORACLE_BASE}/flash_recovery_area"/>
         <initParam name="db_recovery_file_dest_size" value="4096" unit="MB"/>
         <initParam name="open_cursors" value="300"/>
         <initParam name="pga_aggregate_target" value="500" unit="MB"/>
         <initParam name="processes" value="300"/>
         <initParam name="undo_tablespace" value="UNDOTBS1"/>
         <initParam name="sga_max_size" value="2048" unit="MB"/>
         <initParam name="sga_target" value="2048" unit="MB"/>
         <initParam name="optimizer_index_caching" value="95"/>
         <initParam name="optimizer_index_cost_adj" value="10"/>
      </InitParams>
      <MiscParams>
         <databaseType>MULTIPURPOSE</databaseType>
         <maxUserConn>20</maxUserConn>
         <percentageMemTOSGA>40</percentageMemTOSGA>
         <customSGA>true</customSGA>
         <archiveLogMode>false</archiveLogMode>
         <initParamFileName>{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/pfile/init.ora</initParamFileName>
      </MiscParams>
      <SPfile useSPFile="true">{ORACLE_HOME}/dbs/spfile{SID}.ora</SPfile>
   </InitParamAttributes>
   <StorageAttributes>
      <ControlfileAttributes id="Controlfile">
         <maxDatafiles>100</maxDatafiles>
         <maxLogfiles>16</maxLogfiles>
         <maxLogMembers>3</maxLogMembers>
         <maxLogHistory>1</maxLogHistory>
         <maxInstances>8</maxInstances>
         <image name="control01.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
         <image name="control02.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
         <image name="control03.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </ControlfileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/idata01.dbf">
         <tablespace>IDATA</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">500</size>
         <reuse>false</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="KB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/ilog01.dbf">
         <tablespace>ILOG</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">50</size>
         <reuse>false</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="KB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/indx01.dbf">
         <tablespace>INDX</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">50</size>
         <reuse>false</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="KB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf">
         <tablespace>SYSAUX</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">120</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf">
         <tablespace>SYSTEM</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">400</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf">
         <tablespace>TEMP</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">50</size>
         <reuse>true</reuse>
         <autoExtend>false</autoExtend>
         <increment unit="KB">640</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf">
         <tablespace>UNDOTBS1</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">500</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <TablespaceAttributes id="IDATA">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">-1</initSize>
         <increment unit="KB">-1</increment>
         <incrementPercent>1</incrementPercent>
         <minExtends>-1</minExtends>
         <maxExtends>-2</maxExtends>
         <minExtendsSize unit="KB">-1</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/idata01.dbf">
               <id>1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="ILOG">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">-1</initSize>
         <increment unit="KB">-1</increment>
         <incrementPercent>1</incrementPercent>
         <minExtends>-1</minExtends>
         <maxExtends>-2</maxExtends>
         <minExtendsSize unit="KB">-1</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/ilog01.dbf">
               <id>1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="INDX">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">-1</initSize>
         <increment unit="KB">-1</increment>
         <incrementPercent>1</incrementPercent>
         <minExtends>-1</minExtends>
         <maxExtends>-2</maxExtends>
         <minExtendsSize unit="KB">-1</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/indx01.dbf">
               <id>1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="SYSAUX">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="SYSTEM">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>3</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>-1</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="TEMP">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>true</temporary>
         <defaultTemp>true</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>0</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>0</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="UNDOTBS1">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>true</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">512</initSize>
         <increment unit="KB">512</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>8</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">512</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <RedoLogGroupAttributes id="1">
         <reuse>false</reuse>
         <fileSize unit="KB">102400</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo01.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
      <RedoLogGroupAttributes id="2">
         <reuse>false</reuse>
         <fileSize unit="KB">102400</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo02.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
      <RedoLogGroupAttributes id="3">
         <reuse>false</reuse>
         <fileSize unit="KB">102400</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo03.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
   </StorageAttributes>
</DatabaseTemplate>


Teamcenter_Oracle_multitenant.dbt 

<DatabaseTemplate name="tc" description="" version="12.1.0.2.0">
  <CommonAttributes>
    <option name="OMS" value="false"/>
    <option name="JSERVER" value="false"/>
    <option name="SPATIAL" value="false"/>
    <option name="IMEDIA" value="false"/>
    <option name="XDB_PROTOCOLS" value="false">
      <tablespace id="SYSAUX"/>
    </option>
    <option name="ORACLE_TEXT" value="false">
      <tablespace id="SYSAUX"/>
    </option>
    <option name="SAMPLE_SCHEMA" value="false"/>
    <option name="CWMLITE" value="false">
      <tablespace id="SYSAUX"/>
    </option>
    <option name="APEX" value="false">
      <tablespace id="SYSAUX"/>
    </option>
    <option name="DV" value="false">
      <tablespace id="SYSAUX"/>
    </option>
    <option name="NET_EXTENSIONS" value="false"/>
   </CommonAttributes>
   <Variables/>
   <CustomScripts Execute="false"/>
   <InitParamAttributes>
      <InitParams>
        <initParam name="db_name" value="tc"/>
        <initParam name="instance_name" value="tc"/>
        <initParam name="db_domain" value=""/>
        <initParam name="compatible" value="12.1.0.2.0"/>
        <initParam name="enable_pluggable_database" value="true"/>
        <initParam name="pga_aggregate_target" value="500" unit="MB"/>
        <initParam name="sga_max_size" value="2048" unit="MB"/>
        <initParam name="sga_target" value="2048" unit="MB"/>
        <initParam name="optimizer_index_cost_adj" value="10"/>
        <initParam name="optimizer_index_caching" value="95"/>
        <initParam name="processes" value="300"/>
        <initParam name="open_cursors" value="300"/>
        <initParam name="db_file_multiblock_read_count" value="32"/>
        <initParam name="db_recovery_file_dest" value="{ORACLE_BASE}/flash_recovery_area"/>
        <initParam name="db_recovery_file_dest_size" value="6000" unit="MB"/>
        <initParam name="undo_tablespace" value="UNDOTBS1"/>
        <initParam name="control_files" value="(&quot;{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control01.ctl&quot;, &quot;{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control02.ctl&quot;, &quot;{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control03.ctl&quot;)"/>
      </InitParams>
      <MiscParams>
         <databaseType>MULTIPURPOSE</databaseType>
         <maxUserConn>20</maxUserConn>
         <percentageMemTOSGA>40</percentageMemTOSGA>
         <customSGA>true</customSGA>
         <archiveLogMode>false</archiveLogMode>
         <dataVaultEnabled>false</dataVaultEnabled>
         <initParamFileName>{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/pfile/init.ora</initParamFileName>
      </MiscParams>
      <SPfile useSPFile="true">{ORACLE_HOME}/dbs/spfile{SID}.ora</SPfile>
   </InitParamAttributes>
   <StorageAttributes>
      <ControlfileAttributes id="Controlfile">
         <maxDatafiles>100</maxDatafiles>
         <maxLogfiles>16</maxLogfiles>
         <maxLogMembers>3</maxLogMembers>
         <maxLogHistory>1</maxLogHistory>
         <maxInstances>8</maxInstances>
         <image name="control01.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
         <image name="control02.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
         <image name="control03.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </ControlfileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf" con_id="1">
         <tablespace>SYSAUX</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">120</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf" con_id="1">
         <tablespace>SYSTEM</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">400</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf" con_id="1">
         <tablespace>TEMP</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">50</size>
         <reuse>true</reuse>
         <autoExtend>false</autoExtend>
         <increment unit="KB">640</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf" con_id="1">
         <tablespace>UNDOTBS1</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">500</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <TablespaceAttributes id="SYSAUX" con_id="1">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="SYSTEM" con_id="1">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>3</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>-1</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="TEMP" con_id="1">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>true</temporary>
         <defaultTemp>true</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>0</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>0</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="UNDOTBS1" con_id="1">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>true</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">512</initSize>
         <increment unit="KB">512</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>8</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">512</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <autoSegmentMgmt>true</autoSegmentMgmt>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf">
               <id>-1</id>
            </TablespaceDatafileAttributes>
         </datafilesList>
      </TablespaceAttributes>
      <RedoLogGroupAttributes id="1">
         <reuse>false</reuse>
         <fileSize unit="KB">102400</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo01.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
      <RedoLogGroupAttributes id="2">
         <reuse>false</reuse>
         <fileSize unit="KB">102400</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo02.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
      <RedoLogGroupAttributes id="3">
         <reuse>false</reuse>
         <fileSize unit="KB">102400</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo03.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
   </StorageAttributes>
</DatabaseTemplate>






DBCA Template SCREEN SHOT :


















Database and clusterware status:

srv1:/u01/app/oracle/product/12.1.0/dbhome_1 # crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.asm
               ONLINE  ONLINE       srv1                     Started,STABLE
               ONLINE  ONLINE       srv2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
ora.ons
               ONLINE  ONLINE       srv1                     STABLE
               ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       srv2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       srv1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       srv1                     169.254.185.74,STABL
                                                             E
ora.cvu
      1        ONLINE  ONLINE       srv1                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       srv1                     Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       srv2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.smrplm.db
      1        ONLINE  ONLINE       srv1                     Open,STABLE
      2        ONLINE  ONLINE       srv2                     Open,STABLE
ora.srv1.vip
      1        ONLINE  ONLINE       srv1                     STABLE
ora.srv2.vip
      1        ONLINE  ONLINE       srv2                     STABLE
--------------------------------------------------------------------------------
srv1:/u01/app/oracle/product/12.1.0/dbhome_1 #