Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x10] [PC:0xCE518CE, qerixGetKey()+782] [flags: 0x0, count: 1]

Scenario Preview:

We recently hit error in our production node and test environment . So as per suggestion by support. we implemented the same first one is change on parameter on database and second one is apply patch to resolve bug.

ERRORS:

Error at application level and in sqlplus prompt:

ERROR at line 1:

ORA-03113: end-of-file on communication channel

Process ID: 3388

Session ID: 324 Serial number: 55071


Error in Alert.log 

Fri Apr 24 09:41:23 2020

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x10] [PC:0xCE518CE, qerixGetKey()+782] [flags: 0x0, count: 1]

Errors in file /u01/app/oracle/diag/rdbms/TESTENV/TESTENV/trace/TESTENV_ora_2543.trc  (incident=615):

ORA-07445: exception encountered: core dump [qerixGetKey()+782] [SIGSEGV] [ADDR:0x10] [PC:0xCE518CE] [Address not mapped to object] []

Incident details in: /u01/app/oracle/diag/rdbms/TESTENV/TESTENV/incident/incdir_615/TESTENV_ora_2543_i615.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Fri Apr 24 09:41:25 2020

Errors in file /u01/app/oracle/diag/rdbms/TESTENV/TESTENV/incident/incdir_615/TESTENV_ora_2543_i615.trc:

ORA-10173: Dynamic Sampling time-out error

ORA-07445: exception encountered: core dump [qerixGetKey()+782] [SIGSEGV] [ADDR:0x10] [PC:0xCE518CE] [Address not mapped to object] []

Fri Apr 24 09:41:25 2020

Errors in file /u01/app/oracle/diag/rdbms/TESTENV/TESTENV/incident/incdir_615/TESTENV_ora_2543_i615.trc:

ORA-10173: Dynamic Sampling time-out error

ORA-07445: exception encountered: core dump [qerixGetKey()+782] [SIGSEGV] [ADDR:0x10] [PC:0xCE518CE] [Address not mapped to object] []


Solution:


1) The recommended temporary workaround is to set one of the following

       OPTIMIZER_ADAPTIVE_FEATURES=false

     alter session set "_projection_pushdown"=false;

    alter session set optimizer_adaptive_features=false;


  2) The permanent solution is for Teamcenter customers who have downloaded

    and installed Oracle 12.1.0.2 to download and install the following

    patch bundle, which includes the correction for Oracle Bug 18430870.


    This patch is available on the GTAC download server for each

    supported Teamcenter platform in the Oracle / Product Updates / Oracle12

    folder.

     p22809813_121020_MSWIN-x86-64.zip

    p22899531_121020_Linux-x86-64.zip

    p22899531_121020_Solaris86-64.zip

Practice Solution Scenario:

NAME      INSTANCE_NAME    DB_UNIQUE_NAME                 OPEN_MODE            DATABASE_ROLE    CURRENT_SCN        FAILOVER_SCN

--------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

xxxTESTE  TESTENV       TESTENV                     READ WRITE           PRIMARY          NO                 0



SQL> SELECT /*+ DYNAMIC_SAMPLING(t_02 4)*/ ;

SELECT /*+ DYNAMIC_SAMPLING(t_02 4)*/) ) ) ) )

*

ERROR at line 1:

ORA-03113: end-of-file on communication channel

Process ID: 3388

Session ID: 324 Serial number: 55071



SQL>



SQL> ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES = FALSE SCOPE=BOTH;



System altered.



SQL> ALTER SYSTEM SET "_projection_pushdown"=false SCOPE=BOTH;



System altered.



SQL> @d



NAME      INSTANCE_NAME    DB_UNIQUE_NAME                 OPEN_MODE            DATABASE_ROLE    CURRENT_SCN        FAILOVER_SCN

--------- ---------------- ------------------------------ -------------------- ---------------- ------------------ ----------------------------------------

xxxPATCH  TESTENV        TESTENV                      READ WRITE           PRIMARY          NO                 0



SQL> SELECT /*+ DYNAMIC_SAMPLING(t_02 4)*xxxxx -------------------------------;

COLUMN_ENTRY

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

xxx

xxxxx

SQL> sho parameter OPTIMIZER_ADAPTIVE_FEATURES



NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

optimizer_adaptive_features          boolean     FALSE

SQL> sho parameter _projection_pushdown



NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

_projection_pushdown                 boolean     FALSE

SQL>





 

RAC - ORA-15025: could not open disk |WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Scenario : 

Recently isntalling 2 node RAC 12c R1 , We face error during running ./dbca , Although we successfully installed grid setup ,asmca disk configuration  and oracle rdbms binary. but while installing instance we face error. I have done implementation several times but some silly mistake by me.. As was just test so i was taking it seriously fed up installing RAC setup , But today i learn something new for this error. following  (Doc ID 1378747.1)

Error:

ORA-15025: could not open disk "/dev/oracleasm/disks/FRA"

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so


alert log error on rac1.log


ORA-15025: could not open disk "/dev/oracleasm/disks/FRA"

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Wed Apr 22 14:38:18 2020
NOTE: ASMB connected to ASM instance +ASM1 osid: 15845 (Flex mode; client id 0xffffffffffffffff)
NOTE: initiating MARK startup
Starting background process MARK
Wed Apr 22 14:38:18 2020
MARK started with pid=35, OS id=15849
Wed Apr 22 14:38:19 2020
NOTE: MARK has subscribed
Wed Apr 22 14:38:26 2020
NOTE: ASMB mounting group 3 (FRA)
Wed Apr 22 14:38:28 2020
ORA-15025: could not open disk "/dev/oracleasm/disks/FRA"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15183: ASMLIB initialization error [driver/agent not installed]
NOTE: Disk 0 in group 3 could not be opened.
WARNING: Failed to complete group 3
WARNING: group 3 is being dismounted.
WARNING: ASMB force dismounting group 3 (FRA) due to failed mount
SUCCESS: diskgroup FRA was dismounted
NOTE: ASMB mounting group 2 (DATA)
Wed Apr 22 14:38:32 2020
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15040: diskgroup is incomplete
NOTE: Disk 0 in group 2 could not be opened.
WARNING: Failed to complete group 2
WARNING: group 2 is being dismounted.
WARNING: ASMB force dismounting group 2 (DATA) due to failed mount
SUCCESS: diskgroup DATA was dismounted
NOTE: ASMB mounting group 2 (DATA)
Wed Apr 22 14:38:33 2020
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15040: diskgroup is incomplete
NOTE: Disk 0 in group 2 could not be opened.
WARNING: Failed to complete group 2
WARNING: group 2 is being dismounted.

############################################################################

Solution:

After troubleshooting i checked  asmdba group was not provided to oracle and grid user. Although created but some how missed to proved group on user.


grid@sgdcplm01:~> id
uid=54323(grid) gid=54321(oinstall) groups=54321(oinstall),54329(asmadmin)
grid@sgdcplm01:~> su oracle
Password:
oracle@sgdcplm01:/home/grid> id
uid=502(oracle) gid=54321(oinstall) groups=54321(oinstall),503(dba),504(oper),1000(beoper) ,54329(asmadmin)
oracle@sgdcplm01:/home/grid>

( Adding group to oracle and grid user by executing from root user )

 # usermod -a -G asmdba oracle

# usermod -a -G asmdba grid

login as: grid

Keyboard-interactive authentication prompts from server:

| Password:

End of keyboard-interactive prompts from server

Last login: Wed Apr 22 09:54:57 2020 from 10.4.6.143

grid@sgdcplm01:~> id

uid=54323(grid) gid=54321(oinstall) groups=54321(oinstall),54327(asmdba),54329(asmadmin)

grid@sgdcplm01:~> su oracle

Password:

oracle@sgdcplm01:/home/grid> id

uid=502(oracle) gid=54321(oinstall) groups=54321(oinstall),503(dba),504(oper),1000(beoper),54327(asmdba),54329(asmadmin)

oracle@sgdcplm01:/home/grid>


grid@srv2:/u01/app/12.1.0/grid/bin> . oraenv
ORACLE_SID = [grid] ? +ASM2
The Oracle base has been set to /u01/app/grid

grid@srv2:/u01/app/12.1.0/grid/bin> ./setasmgidwrapo=/u01/app/oracle/product/12.1.0/db_1/bin/oracle


grid@srv2:/u01/app/12.1.0/grid/bin>

(NOTE: Following doc id below: Database Will Not Mount: ORA-15025, ORA-27041, 'Permission denied', ORA-15081 (Doc ID 1378747.1)

Alert log :

Wed Apr 22 15:54:45 2020
NOTE: ASMB registering with ASM instance as Standard client 0xffffffffffffffff (reg:2301527851) (new connection)
Starting background process RBAL
Wed Apr 22 15:54:45 2020
RBAL started with pid=34, OS id=11971
Wed Apr 22 15:54:45 2020
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Wed Apr 22 15:54:45 2020
NOTE: ASMB connected to ASM instance +ASM1 osid: 11969 (Flex mode; client id 0xffffffffffffffff)
NOTE: initiating MARK startup
Starting background process MARK
Wed Apr 22 15:54:45 2020
MARK started with pid=35, OS id=11973
Wed Apr 22 15:54:45 2020
NOTE: MARK has subscribed
Wed Apr 22 15:54:50 2020
NOTE: ASMB mounting group 3 (FRA)
NOTE: Assigning number (3,0) to disk (/dev/oracleasm/disks/FRA)
SUCCESS: mounted group 3 (FRA)
NOTE: grp 3 disk 0: FRA_0000 path:/dev/oracleasm/disks/FRA
NOTE: ASMB mounting group 2 (DATA)
NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/DATA)
SUCCESS: mounted group 2 (DATA)
NOTE: grp 2 disk 0: DATA_0000 path:/dev/oracleasm/disks/DATA

RAC : [INS-13001] Environment does not meet minimum requirements.

Unzip grid software and go to directory /u01/grid/stage/cvu/cv/admin andmake changes on cvu_config file
srv1:/u01/grid # ll
total 24
drwxrwxr-x 1 grid oinstall  298 Apr 21 20:53 install
drwxrwxr-x 1 grid oinstall   32 Jul  7  2014 response
drwxrwxr-x 1 grid oinstall   40 Jul  7  2014 rpm
-rwxrwxr-x 1 grid oinstall 8534 Jul  7  2014 runInstaller
-rwxrwxr-x 1 grid oinstall 5085 Dec 20  2013 runcluvfy.sh
drwxrwxr-x 1 grid oinstall   30 Jul  7  2014 sshsetup
drwxrwxr-x 1 grid oinstall  910 Jul  7  2014 stage
-rwxrwxr-x 1 grid oinstall  500 Feb  7  2013 welcome.html
srv1:/u01/grid #

srv1:/u01/grid/stage/cvu/cv/admin # ll
total 62472
-rwxrwxr-x 1 grid oinstall     1263 Jul  1  2014 cvu_config
-rwxrwxr-x 1 grid oinstall 63964254 Jul  1  2014 cvunetquery
srv1:/u01/grid/stage/cvu/cv/admin #

# Enable value 

CV_ASSUME_DISTID=SUSE11

zypper install libstdc++46-devel libstdc++33 libstdc++33-devel libelf-devel gcc-c++ libaio-devel sysstat unixODBC unixODBC-devel xdpyinfo

RAC [INS-06003] Failed to setup passwordless SSH connectivity with the following node(s)

• Scenario Preview : We recently find ssh error on during installation. As per my experience only found this error only on open suse 12 server.

• OpenSSH_6.7: ERROR [INS-06003] Failed to setup passwordless SSH connectivity During Grid Infrastructure Install (Doc ID 2111092.1)

• srv1:/u01/grid/sshsetup # vi  /etc/ssh/sshd_config (both nodes)

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1   

(Doc ID 2111092.1)

• srv1:/u01/grid/sshsetup # service sshd restart (both nodes)
• srv1:/u01/grid/sshsetup # service sshd status

sshd.service - OpenSSH Daemon

   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)

   Active: active (running) since Tue 2020-04-21 11:28:18 IST; 20s ago

  Process: 6035 ExecStartPre=/usr/sbin/sshd -t $SSHD_OPTS (code=exited, status=0/SUCCESS)

  Process: 6032 ExecStartPre=/usr/sbin/sshd-gen-keys-start (code=exited, status=0/SUCCESS)

 Main PID: 6039 (sshd)

    Tasks: 1

   CGroup: /system.slice/sshd.service

           └─6039 /usr/sbin/sshd -D

Apr 21 11:28:18 srv1 systemd[1]: Stopped OpenSSH Daemon.

Apr 21 11:28:18 srv1 systemd[1]: Starting OpenSSH Daemon...

Apr 21 11:28:18 srv1 sshd-gen-keys-start[6032]: Checking for missing server keys in /etc/ssh

Apr 21 11:28:18 srv1 sshd[6039]: Server listening on 0.0.0.0 port 22.

Apr 21 11:28:18 srv1 sshd[6039]: Server listening on :: port 22.

Apr 21 11:28:18 srv1 systemd[1]: Started OpenSSH Daemon.

srv1:/u01/grid/sshsetup #

RAC _Patch _Failed_scenario : CRS-6706: Oracle Clusterware Release patch level ('966527961') does not match Software patch level ('2919480821'). Oracle Clusterware cannot be started.

Scenario Preview : Recently with got error on TEST 2 node grid 12c  regarding unable to create ACFS volume.

• Got Error - ORA-15032: not all alterations performed || ORA-15477: cannot communicate with the volume driver

• when we apply patch got error ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5' after searching trace and founding this is due to bugs as per Doc ID 2205623.1
• ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.10.0-514.el7.x86_64' (Doc ID 2205623.1)

• So we find out  this error  due to bug 25078431. So we apply patch below forcefully. 

• Patch applied successfully but cluster ware didn't start so we have taken grid, database and Orainventory backup at the tar level , so unzip grid and database binary and using  Rlink (Doc ID 1536057.1) 
•  started the services and cluster service started. Meanwhile on node 1 remains Active as we applied on 2 node so service are running on node1.

Please find example scenario below:

suse2:/u01 # export ORACLE_HOME=/u01/app/12.1.0/grid
suse2:/u01 # export PATH=$ORACLE_HOME/OPatch:$PATH:$ORACLE_HOME/bin
suse2:/u01 # export PATH=$ORACLE_HOME/perl/bin:$PATH

suse2:/u01/25078431 # opatchauto apply -analyze

OPatchauto session is initiated at Fri Apr 10 16:36:02 2020

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-04-10_04-36-09PM.log.


Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2020-04-10_04-42-21PM.log
The id for this session is 8Q6I

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/dbhome_1

Patch applicability verified successfully on home /u01/app/12.1.0/grid


Verifying SQL patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:suse2
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Version:12.1.0.2.0


==Following patches were SKIPPED:

Patch: /u01/25078431/25078431
Reason: This patch is not applicable to this specified target type - "rac_database"


Host:suse2
CRS Home:/u01/app/12.1.0/grid
Version:12.1.0.2.0


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /u01/25078431/25078431
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-10_16-42-36PM_1.log



OPatchauto session completed at Fri Apr 10 16:42:42 2020
Time taken to complete the session 6 minutes, 41 seconds
suse2:/u01/25078431 #


suse2:/u01/25078431/25078431 # su grid
grid@suse2:/u01/25078431/25078431> export ORACLE_HOME=/u01/app/12.1.0/grid
grid@suse2:/u01/25078431/25078431> export PATH=$ORACLE_HOME/OPatch:$PATH:$ORACLE_HOME/bin
grid@suse2:/u01/25078431/25078431> export PATH=$ORACLE_HOME/perl/bin:$PATH
grid@suse2:/u01/25078431/25078431>
grid@suse2:/u01/25078431/25078431> /u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local /u01/25078431/25078431
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.1.0.2.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2020-04-10_16-48-57PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   25078431

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/12.1.0/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '25078431' to OH '/u01/app/12.1.0/grid'

Patching component oracle.usm, 12.1.0.2.0...
Patch 25078431 successfully applied.
Sub-set patch [24007012] has become inactive due to the application of a super-set patch [25078431].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2020-04-10_16-48-57PM_1.log

OPatch succeeded.
grid@suse2:/u01/25078431/25078431>



suse2:/u01/25078431/25078431 # . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/grid
suse2:/u01/25078431/25078431 # crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('966527961') does not match Software patch level ('2919480821'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

suse2:/u01/25078431/25078431 # ps -ef |grep pmon
root      9224  9126  0 16:56 pts/0    00:00:00 grep --color=auto pmon
suse2:/u01/25078431/25078431 # cd /u01/app/grid/
suse2:/u01/app/grid # ll


suse2:/u01/app/12.1.0/grid/crs/install # ./rootcrs.sh -patch
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2020/04/10 17:21:56 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2020/04/10 17:24:16 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

suse2:/u01/app/12.1.0/grid/crs/install # ./roothas.pl -postpatch
Using configuration parameter file: ./crsconfig_params
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

suse2:/u01/app/12.1.0/grid/crs/install # ps -ef |grep d.bin
root       632     1  0 21:16 ?        00:00:00 /usr/lib/wicked/bin/wickedd-auto4 --systemd --foreground
root       634     1  0 21:16 ?        00:00:00 /usr/lib/wicked/bin/wickedd-dhcp6 --systemd --foreground
root       636     1  0 21:16 ?        00:00:00 /usr/lib/wicked/bin/wickedd-dhcp4 --systemd --foreground
root      1754     1  1 21:16 ?        00:00:09 /u01/app/12.1.0/grid/bin/ohasd.bin reboot
root      1905     1  0 21:17 ?        00:00:03 /u01/app/12.1.0/grid/bin/orarootagent.bin
grid      1972     1  0 21:17 ?        00:00:01 /u01/app/12.1.0/grid/bin/oraagent.bin
grid      1984     1  0 21:17 ?        00:00:01 /u01/app/12.1.0/grid/bin/mdnsd.bin
grid      1987     1  0 21:17 ?        00:00:02 /u01/app/12.1.0/grid/bin/evmd.bin
grid      2001     1  0 21:17 ?        00:00:01 /u01/app/12.1.0/grid/bin/gpnpd.bin
grid      2025     1  0 21:17 ?        00:00:02 /u01/app/12.1.0/grid/bin/gipcd.bin
grid      2042  1987  0 21:17 ?        00:00:00 /u01/app/12.1.0/grid/bin/evmlogger.bin -o /u01/app/12.1.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/12.1.0/grid/log/[HOSTNAME]/evmd/evmlogger.log
root      2162     1  0 21:17 ?        00:00:01 /u01/app/12.1.0/grid/bin/cssdmonitor
root      2180     1  0 21:17 ?        00:00:01 /u01/app/12.1.0/grid/bin/cssdagent
grid      2193     1  0 21:17 ?        00:00:03 /u01/app/12.1.0/grid/bin/ocssd.bin
root      2272     1  0 21:17 ?        00:00:03 /u01/app/12.1.0/grid/bin/octssd.bin reboot
root      6734  5388  0 21:28 pts/0    00:00:00 grep --color=auto d.bin

suse2:/u01/app/12.1.0/grid/crs/install # crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager


suse2:/u01/app/12.1.0/grid/crs/install # crsctl stop crs

NOTE: We found on crs alert log ctssd| ocssd time is different between 2 nodes , So we correct data and time and started the services but no solution found, done troubleshooting also but unable to start clusterware service on node 2. rebooted also. 

suse2:/u01/app/12.1.0/grid/crs/install # ./roothas.pl -postpatch
Using configuration parameter file: ./crsconfig_params
CLSU-00105: operating system interface has reported an internal failure
CLSU-00103: error location: canexec2
CLSU-00104: additional error information: no exe permission, file [/u01/app/12.1.0/grid/bin/ohasd]



CRS ALERT.log


2020-04-10 18:06:26.735 [OCSSD(2193)]CRS-1652: Starting clean up of CRSD resources.
2020-04-10 18:06:26.746 [OCSSD(2193)]CRS-1653: The clean up of the CRSD resources failed.
Fri Apr 10 18:06:26 2020
Errors in file /u01/app/grid/diag/crs/suse2/crs/trace/ocssd.trc  (incident=9):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/grid/diag/crs/suse2/crs/incident/incdir_9/ocssd_i9.trc

2020-04-10 18:06:27.014 [OCSSD(2193)]CRS-8503: Oracle Clusterware OCSSD process with operating system process ID 2193 experienced fatal signal or exception code 6
Sweep [inc][9]: completed
2020-04-10 18:06:27.387 [ORAROOTAGENT(1905)]CRS-5013: Agent "ORAROOTAGENT" failed to start process "/u01/app/12.1.0/grid/bin/octssd" for action "start": details at "(:CLSN00008:)" in "/u01/app/grid/diag/crs/suse2/crs/trace/ohasd_orarootagent_root.trc"
2020-04-10 18:06:27.400 [OHASD(1754)]CRS-2878: Failed to restart resource 'ora.ctssd'
2020-04-14 22:26:52.525 [CLSECHO(3762)]CRS-10001: 14-Apr-20 22:26 ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5'
2020-04-14 22:26:52.617 [CLSECHO(3765)]CRS-10001: 14-Apr-20 22:26 ACFS-9201: Not Supported
2020-04-14 22:26:53.120 [CLSCFG(3767)]CRS-1810: Node-specific configuration for node suse2 in Oracle Local Registry was patched to patch level 966527961.
2020-04-14 22:34:14.217 [CLSECHO(8009)]CRS-10001: 14-Apr-20 22:34 ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5'
2020-04-14 22:34:14.280 [CLSECHO(8011)]CRS-10001: 14-Apr-20 22:34 ACFS-9201: Not Supported
2020-04-14 22:34:14.354 [CLSCFG(8013)]CRS-1810: Node-specific configuration for node suse2 in Oracle Local Registry was patched to patch level 966527961.


2020-04-10 18:24:16.122 [CLSECHO(9279)]CRS-10001: 10-Apr-20 18:24 ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5'
2020-04-10 18:24:16.133 [CLSECHO(9281)]CRS-10001: 10-Apr-20 18:24 ACFS-9201: Not Supported
2020-04-10 18:24:16.236 [CLSCFG(9283)]CRS-1810: Node-specific configuration for node suse2 in Oracle Local Registry was patched to patch level 966527961.

2020-04-10 19:35:48.037 [OHASD(25258)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 25258
2020-04-10 19:35:48.037 [OHASD(25258)]CRS-0715: Oracle High Availability Service has timed out waiting for init.ohasd to be started.
2020-04-10 19:39:10.788 [OHASD(26174)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 26174
2020-04-10 19:39:10.788 [OHASD(26174)]CRS-0715: Oracle High Availability Service has timed out waiting for init.ohasd to be started.
2020-04-10 19:41:18.982 [OHASD(26687)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 26687
2020-04-10 19:41:18.982 [OHASD(26687)]CRS-0715: Oracle High Availability Service has timed out waiting for init.ohasd to be started.


( Try to rollback patch but no luck )

use2:/u01/25078431/25078431 # . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/grid
suse2:/u01/25078431/25078431 # export ORACLE_HOME=/u01/app/12.1.0/grid
suse2:/u01/25078431/25078431 # export PATH=$ORACLE_HOME/OPatch:$PATH:$ORACLE_HOME/bin
suse2:/u01/25078431/25078431 # export PATH=$ORACLE_HOME/perl/bin:$PATH
suse2:/u01/25078431/25078431 # opatchauto rollback /u01/25078431/25078431 -oh /u01/app/12.1.0/grid

OPatchauto session is initiated at Mon Apr 20 13:36:29 2020

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-04-20_01-36-35PM.log.

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2020-04-20_01-39-31PM.log
The id for this session is VZM8
OPATCHAUTO-72132: Grid is not running on the local host.
OPATCHAUTO-72132: Cannot start a new apply or rollback session when the local grid is not running.
OPATCHAUTO-72132: Please start grid service on the local host to start patching.
OPatchAuto failed.

OPatchauto session completed at Mon Apr 20 13:39:41 2020
Time taken to complete the session 3 minutes, 12 seconds

 opatchauto failed with error code 42
suse2:/u01/25078431/25078431 # cd /u01/app/12.1.0/grid/




########################    Solution   ##############


Unzip  node 2 grid and database binary :

• tar zvxf grid.tar.gz

•  tar zvxf 12.1.0.tar.gz

• Perform  Rlink (Doc ID 1536057.1) 
• reboot

################# STATUS AFTER Reboot ################

suse2:~ # date -s "13 APR 2020 20:23:10"

Mon Apr 13 20:23:10 IST 2020

suse2:~ # ps -ef |grep pmon

grid      2615     1  0 19:59 ?        00:00:00 asm_pmon_+ASM2

oracle    3252     1  0 20:00 ?        00:00:00 ora_pmon_rac2

root     11213  9425  0 20:23 pts/0    00:00:00 grep --color=auto pmon

suse2:~ # crsctl status resource -t

If 'crsctl' is not a typo you can use command-not-found to lookup the package that contains it, like this:

    cnf crsctl

suse2:~ # . oraenv

ORACLE_SID = [root] ? +ASM2

The Oracle base has been set to /u01/app/grid

suse2:~ # crsctl status resource -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ACFS.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.CRS.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.DATA.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.FRA.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.asm

               ONLINE  ONLINE       suse1                    Started,STABLE

               ONLINE  ONLINE       suse2                    Started,STABLE

ora.net1.network

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.ons

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       suse2                    STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       suse1                    STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       suse1                    STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       suse1                    169.254.74.130 192.1

                                                             68.10.5,STABLE

ora.cvu

      1        ONLINE  ONLINE       suse1                    STABLE

ora.mgmtdb

      1        ONLINE  ONLINE       suse1                    Open,STABLE

ora.oc4j

      1        ONLINE  ONLINE       suse1                    STABLE

ora.rac.db

      1        ONLINE  ONLINE       suse1                    Open,STABLE

      2        ONLINE  ONLINE       suse2                    Open,STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       suse2                    STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       suse1                    STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       suse1                    STABLE

ora.suse1.vip

      1        ONLINE  ONLINE       suse1                    STABLE

ora.suse2.vip

      1        ONLINE  ONLINE       suse2                    STABLE

--------------------------------------------------------------------------------

suse2:~ # crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'suse2'

CRS-2673: Attempting to stop 'ora.crsd' on 'suse2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'suse2'

CRS-2673: Attempting to stop 'ora.CRS.dg' on 'suse2'

CRS-2673: Attempting to stop 'ora.rac.db' on 'suse2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'suse2'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'suse2'

CRS-2677: Stop of 'ora.CRS.dg' on 'suse2' succeeded

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'suse2'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.suse2.vip' on 'suse2'

CRS-2677: Stop of 'ora.scan1.vip' on 'suse2' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'suse1'

CRS-2677: Stop of 'ora.rac.db' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'suse2'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'suse2'

CRS-2677: Stop of 'ora.suse2.vip' on 'suse2' succeeded

CRS-2672: Attempting to start 'ora.suse2.vip' on 'suse1'

CRS-2677: Stop of 'ora.DATA.dg' on 'suse2' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.ACFS.dg' on 'suse2'

CRS-2677: Stop of 'ora.ACFS.dg' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'suse2'

CRS-2677: Stop of 'ora.asm' on 'suse2' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'suse1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'suse1'

CRS-2676: Start of 'ora.suse2.vip' on 'suse1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'suse1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'suse2'

CRS-5014: Agent "ORAAGENT" timed out starting process "/u01/app/12.1.0/grid/opmn/bin/onsctli" for action "stop": details at "(:CLSN00009:)" in "/u01/app/grid/diag/crs/suse2/crs/trace/crsd_oraagent_grid.trc"

CRS-2675: Stop of 'ora.ons' on 'suse2' failed

CRS-2679: Attempting to clean 'ora.ons' on 'suse2'

CRS-2681: Clean of 'ora.ons' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'suse2'

CRS-2677: Stop of 'ora.net1.network' on 'suse2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'suse2' has completed

CRS-2677: Stop of 'ora.crsd' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'suse2'

CRS-2673: Attempting to stop 'ora.evmd' on 'suse2'

CRS-2673: Attempting to stop 'ora.storage' on 'suse2'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'suse2'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'suse2'

CRS-2677: Stop of 'ora.storage' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'suse2'

CRS-2677: Stop of 'ora.ctssd' on 'suse2' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'suse2' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'suse2' succeeded

CRS-2677: Stop of 'ora.evmd' on 'suse2' succeeded

CRS-2677: Stop of 'ora.asm' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'suse2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'suse2'

CRS-2677: Stop of 'ora.cssd' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'suse2'

CRS-2677: Stop of 'ora.crf' on 'suse2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'suse2'

CRS-2677: Stop of 'ora.gipcd' on 'suse2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'suse2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

suse2:~ #

suse2:~ #

suse2:~ # crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

######################################################################

suse2:/u01/app/12.1.0 # crsctl status resource -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ACFS.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.CRS.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.DATA.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.FRA.dg

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.asm

               ONLINE  ONLINE       suse1                    Started,STABLE

               ONLINE  ONLINE       suse2                    Started,STABLE

ora.net1.network

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

ora.ons

               ONLINE  ONLINE       suse1                    STABLE

               ONLINE  ONLINE       suse2                    STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       suse2                    STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       suse1                    STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       suse1                    STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       suse1                    169.254.74.130 192.1

                                                             68.10.5,STABLE

ora.cvu

      1        ONLINE  ONLINE       suse1                    STABLE

ora.mgmtdb

      1        ONLINE  ONLINE       suse1                    Open,STABLE

ora.oc4j

      1        ONLINE  ONLINE       suse1                    STABLE

ora.rac.db

      1        ONLINE  ONLINE       suse1                    Open,STABLE

      2        ONLINE  ONLINE       suse2                    Open,STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       suse2                    STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       suse1                    STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       suse1                    STABLE

ora.suse1.vip

      1        ONLINE  ONLINE       suse1                    STABLE

ora.suse2.vip

      1        ONLINE  ONLINE       suse2                    STABLE

--------------------------------------------------------------------------------

suse2:/u01/app/12.1.0 #

##################################################################

suse2:~ # su oracle

oracle@suse2:/root> cd

oracle@suse2:~> . oraenv

ORACLE_SID = [oracle] ? rac

The Oracle base has been set to /u01/app/oracle

oracle@suse2:~> sqlplus sys/xxx@rac as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Apr 13 20:25:36 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> !date

Mon Apr 13 20:25:54 IST 2020

SQL> select name,open_mode,database_Role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

RAC       READ WRITE           PRIMARY

SQL> select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,STATUS from v$instance;

INSTANCE_NUMBER INSTANCE_NAME

--------------- ----------------

HOST_NAME                                                        STATUS

---------------------------------------------------------------- ------------

              2 rac2

suse2                                                            OPEN

SQL>

RAC - CRS-2403: The Cluster Time Synchronization Service on host 02 is in observer mode.

Scenario : We have 2 node 12c 12.1.0.2.0 grid clusterware setup ,  Time synchronization service is being managed by clusterware ctssd service.

Some how our linux guys make time sync entry between two nodes with NTP Server without downtime. Due to this CTSS is gone to  Observer state. Switching over to clock synchronization checks using NTP without any downtime. My post motive was  just to share error details.


CRS alert log below:
/u01/app/grid/diag/crs/02/crs/trace/alert.log


2020-04-13 12:58:49.279 [OCTSSD(4195)]CRS-2403: The Cluster Time Synchronization Service on host 02 is in observer mode.
2020-04-13 13:00:09.296 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.
2020-04-13 13:30:10.200 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.

2020-04-13 12:58:47.221 [OCTSSD(1477)]CRS-2403: The Cluster Time Synchronization Service on host 01 is in observer mode.
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
02,01
PRVF-5415 : Check to see if NTP daemon or service is running failed

grid@01:~> . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/grid
grid@01:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

grid@01:~> ssh sgdcplm02
Last login: Tue Apr 14 09:14:25 2020 from 10.4.6.138
grid@02:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.


##############################################################################

grid@01:/u01/app/12.1.0/grid/bin> cluvfy comp clocksync -n all -verbose

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  01                             passed
  02                             passed
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  02                             Observer
  01                             Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
  Node Name                             File exists?
  ------------------------------------  ------------------------
  02                             yes
  01                             yes
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  02                             no
  01                             no
PRVF-7590 : "ntpd" is not running on node "02"
PRVF-7590 : "ntpd" is not running on node "01"
PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed


PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
grid@01:/u01/app/12.1.0/grid/bin> cat /etc/ntp.conf

############################################################################

Alert log after changing to ntp  in octssd.trc

grid@01:/u01/app/grid/diag/crs/01/crs/trace> tail -1000f octssd.trc

2020-04-14 10:27:54.403648 :    CTSS:3791980288: sclsctss_gvss3: NTP active, forcing observer mode
2020-04-14 10:27:54.403654 :    CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
2020-04-14 10:28:06.943670 :    CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xee], offset[0 ms]}, length=[8].
2020-04-14 10:28:09.554924 :GIPCHTHR:3817641728:  gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-14 10:28:19.557075 :GIPCHTHR:3815540480:  gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-14 10:28:24.406977 :    CTSS:3791980288: sclsctss_ivsr1: default config file found


old_octssd.trc

2020-04-12 00:39:02.478503 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478517 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478521 :    CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:07.378919 :    CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:19.984439 :GIPCHTHR:3817641728:  gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-12 00:39:31.991080 :GIPCHTHR:3815540480:  gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:39:32.481366 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481379 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481382 :    CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:37.383441 :    CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:50.988957 :GIPCHTHR:3817641728:  gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-12 00:40:01.992865 :GIPCHTHR:3815540480:  gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:40:02.485423 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:40:02.485450 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found

grid@02:~>crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUP.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.CRS.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.DATA.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.FRA1.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.FRA2.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.asm
               ONLINE  ONLINE       01                Started,STABLE
               ONLINE  ONLINE       02                Started,STABLE
ora.net1.network
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.ons
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       02                STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       01                STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       01                STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       01                169.254.238.75 192.1
                                                             68.0.63,STABLE
ora.cvu
      1        ONLINE  ONLINE       01                STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       01                Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       01                STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       02                STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       01                STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       01                STABLE
ora.01.vip
      1        ONLINE  ONLINE       01                STABLE
ora.02.vip
      1        ONLINE  ONLINE       02                STABLE
ora.rac.acrac.svc
      1        ONLINE  ONLINE       01                STABLE
ora.rac.db
      1        ONLINE  ONLINE       02                Open,STABLE
      2        ONLINE  ONLINE       01                Open,STABLE
ora.rac.pretaf.svc
      1        ONLINE  ONLINE       02                STABLE
ora.rac.pretaf_preconnect.svc
      1        ONLINE  ONLINE       01                STABLE
ora.rac.staf.svc
      1        ONLINE  ONLINE       01                STABLE
--------------------------------------------------------------------------------

RAC - Take TAR level binary backup of grid_home, db_home and orainventory_home

Example : How to take TAR level binary backup of grid_home, db_home and Orainventory_home

• suse2:/u01/app # crsctl stop crs

• suse2:/u01/app # tar -czvf grid.tar.gz ./grid

• suse2:/u01/app # tar -czvf oraInventory.tar.gz ./oraInventory

• suse2:/u01/app/oracle/product # tar -czvf 12.1.0.tar.gz ./12.1.0

( Example : Unzip OS Grid Tar Backup )

• suse2:/u01/app/12.1.0 # ll

total 4051528
drwxr-xr-x 77 root oinstall       4096 Feb 20 17:09 grid
-rw-r--r--  1 root root     4148753092 Apr  9 13:36 grid.tar.gz

• suse2:/u01/app/12.1.0 # mv grid grid_old

• suse2:/u01/app/12.1.0 # ll

total 4051528
-rw-r--r--  1 root root     4148753092 Apr  9 13:36 grid.tar.gz
drwxr-xr-x 77 root oinstall       4096 Feb 20 17:09 grid_old
suse2:/u01/app/12.1.0 #

(Finally Unzipping grid.tar.gz)

• suse2:/u01/app/12.1.0 # tar zvxf grid.tar.gz

drwxr-xr-x 77 root oinstall       4096 Feb 20 17:09 grid
-rw-r--r--  1 root root     4148753092 Apr  9 13:36 grid.tar.gz
suse2:/u01/app/12.1.0 #

ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5'

suse1:/u01/app/12.1.0/grid/bin # ls -ltr acfsload*
-rwxr-xr-x 1 root oinstall 759 Feb 25 19:42 acfsload

suse1:/u01/app/12.1.0/grid/bin # ./acfsload start

./acfsload: line 34: /u01/app/12.1.0/grid/OPatch/lib/acfstoolsdriver.sh: No such file or directory

suse1:/u01/app/12.1.0/grid/bin # /sbin/acfsutil registry
bash: /sbin/acfsutil: No such file or directory



suse1:/u01/app/12.1.0/grid/bin # ./acfsroot install
ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5'
suse1:/u01/app/12.1.0/grid/bin #



grid@suse1:/u01/app/oracle/product/12.1.0/dbhome_1/OPatch> acfsdriverstate supported

ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5'
ACFS-9201: Not Supported



Reference Link for oracle support:

acfsload fails with /lib/acfstoolsdriver.sh: No such file or directory (Doc ID 1908430.1)

Top Issues That Cause ACFS drivers failed to load (Doc ID 2245689.1)

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown' on Linux (Doc ID 1548846.1)

ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.10.0-514.el7.x86_64' (Doc ID 2205623.1)

ACFS BUG PATCH - 21231953

ACFS Error - ORA-15032: not all alterations performed || ORA-15477: cannot communicate with the volume driver

grid@suse2:/u01/app/12.1.0/grid/OPatch> ./opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.1.0.2.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2020-04-08_20-18-59PM_1.log

Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2020-04-08_20-18-59PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: suse2.example.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 12c                                       12.1.0.2.0
There are 1 products installed in this Oracle Home.


Interim patches (4) :

Patch  24007012     : applied on Thu Feb 20 17:16:03 IST 2020
Unique Patch ID:  20511653
Patch description:  "ACFS Patch Set Update : 12.1.0.2.161018 (24007012)"
   Created on 30 Aug 2016, 00:17:17 hrs PST8PDT
   Bugs fixed:
     19060056, 21369858, 19355146, 19690653, 19526234, 20952520, 19475588
     21231953, 18510745, 19149476, 19013966, 22222222, 19859183, 21519796
     19469788, 21555038, 18877486, 20438706, 19610022, 19335268, 21058846
     22472364, 22319558, 19183802, 19452723, 21124596, 19051391, 20010980
     19526283, 22198405, 22689989, 19279106, 19078259, 19644505, 21077218
     21233961, 19184398, 18321597, 19811784, 18185024, 22174238, 20098064
     19353057, 19502657, 22661787, 19434261, 18951113, 20140148, 18900953
     19053182, 22975177, 18955907, 18934139, 19134464, 19450090, 19517835
     22241945, 20923224, 19446719, 20402287, 19201087, 21208140, 19649858
     21076821, 19428756, 19001684, 19610001, 18899600, 20314907, 19509898
     19528981, 18957085, 19127216, 23557970, 19901825, 22810422, 18915417
     19270227, 22087780, 22198176, 19919907, 19593769, 23112252, 19557156
     19195735, 23252830, 22846612, 20681968, 18851012, 19844362

Patch  24006101     : applied on Thu Feb 20 17:10:25 IST 2020
Unique Patch ID:  20683584
Patch description:  "Database Patch Set Update : 12.1.0.2.161018 (24006101)"
   Created on 1 Oct 2016, 12:33:50 hrs PST8PDT
Sub-patch  23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)"
Sub-patch  22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)"
Sub-patch  21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
Sub-patch  21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch  20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch  20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch  19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
   Bugs fixed:
     19309466, 19902195, 18250893, 17655240, 19028800, 21266085, 19035573
     19366375, 18456643, 19289642, 18845653, 19915271, 21291274, 18417036
     20475845, 22528741, 22148226, 19243521, 19658708, 21153266, 19326908
     19597583, 17414008, 20897759, 19174430, 22046677, 22243719, 21960504
     19054077, 20688221, 22657942, 21899588, 20428621, 13542050, 21387964
     19723336, 19835133, 17532734, 19333670, 21373473, 23260854, 19687159
     14643995, 20977794, 19012119, 20734332, 19869255, 17551063, 18681056
     19676012, 20588502, 19520602, 19841800, 19439759, 20245930, 19303936
     19001359, 19393542, 22916353, 21099555, 22353346, 19141838, 19644859
     21915719, 21421886, 19908836, 19358317, 19134173, 19524158, 20447445
     21188584, 19335438, 19390567, 19058490, 18799993, 19769480, 21097043
     21225209, 19284031, 19450314, 19016730, 18967382, 20919320, 22075064
     20347562, 20348653, 22551446, 19896336, 20048359, 19524384, 16439813
     18354830, 20596234, 20440930, 20936905, 19171086, 17867700, 19791273
     18990023, 21241829, 19591608, 22296366, 20173897, 18914624, 19571367
     17210525, 21260431, 19501299, 20425790, 19124589, 19402853, 19888853
     16870214, 17722075, 18202441, 18743542, 19001390, 19081128, 22173980
     21875360, 19178851, 19149990, 20951038, 22168163, 19606174, 20848335
     19791377, 19050649, 19382851, 20920911, 20528052, 22762046, 19189525
     20598042, 18988834, 19176326, 19048007, 19238590, 20387265, 21263635
     20356733, 22160989, 22380919, 18499088, 18436647, 20825533, 18952989
     20794034, 19468347, 20284155, 19883092, 20657441, 17365043, 20952966
     19176223, 21300341, 18288842, 19699191, 22353199, 22083366, 20669434
     18964978, 19577410, 21373076, 14283239, 19931709, 21239530, 20880215
     20798891, 20043616, 18122373, 19013183, 18856999, 23124895, 21450666
     18893947, 19587324, 20464614, 22062026, 19189317, 21917884, 19708632
     20711718, 20134339, 22815955, 18973548, 22507210, 20101006, 19197175
     18797519, 19597439, 21387128, 19180770, 19879746, 19354335, 19730508
     21785691, 22366558, 6599380, 18964939, 22507234, 20124446, 20627866
     16923858, 19518079, 19371175, 20466322, 18940497, 19074147, 22923409
     17274537, 20842388, 19154375, 20474192, 19044962, 19532017, 19662635
     22374754, 20560611, 18899974, 20471920, 18705806, 19052488, 22024071
     20074391, 21184223, 19157754, 21220620, 18921743, 24316947, 19865345
     19065677, 19065556, 19018447, 19018206, 19777862, 19304354, 22223463
     21322887, 20879889, 20890311, 19578350, 20869721, 21756699, 20217801
     18819908, 19363645, 20898391, 19291380, 20373598, 19248799, 19155797
     19279273, 18886413, 18618122, 19990037, 20922010, 20509482, 18966843
     19077215, 21526048, 20704450, 19068970, 21620471, 19023822, 19670108
     19068610, 20267166, 21756677, 20476175, 18549238, 18674047, 20328248
     19385656, 18849537, 19684504, 20899461, 19308965, 19143550, 19024808
     18948177, 19468991, 20009833, 20868862, 20466628, 21756661, 20397490
     19706965, 22178855, 19604659, 16359751, 19032777, 19329654, 19928926
     20212067, 20603378, 20859910, 19307662, 21847223, 21668627, 20281121
     19075256, 20877664, 19487147, 19076343, 19430401, 24577566, 19676905
     20844426, 20904530, 20925795, 20441797, 21296029, 21629064, 21442094
     23229229, 20708701, 22865673, 19280225, 21315084, 18740837, 20294666
     19989009, 18191823, 21517440, 19174942, 17319928, 19703301, 20122715
     18411216, 20117253, 19258504, 21188532, 24386767, 17890099, 21649497
     16887946, 18791688, 19721304, 22092979, 19490948, 19619732, 20165574
     19272708, 19547370, 20139391, 20228093, 21281532, 19978542, 19409212
     19805359, 19461270, 19434529, 18799063, 18990693, 20378086, 21246723
     20831538, 21632821, 20424899, 20361671, 18674024, 19689979, 20235511
     19873610, 16619249, 21091431, 19440586, 22757364, 18610915, 22175564
     19561643, 19399918, 19195895, 20832516, 20017509, 20907061, 21665897
     20746251, 19183343, 21787056, 21273804, 19067244, 18043064, 21329301
     18885870, 19536415, 17835294, 20446883, 21744290, 20413820, 19185876
     22465352, 20558005, 20402832, 19627012, 20093776, 18909599, 20618595
     16863642, 19639483, 19315691, 20331945, 19512341, 19637186, 19174521
     21479753, 19022470, 18607546, 18306996, 18851894, 19649152, 20581111
     19201867, 20318889, 20898997, 19534363, 19188927

Patch  23854735     : applied on Thu Feb 20 17:07:57 IST 2020
Unique Patch ID:  20511532
Patch description:  "OCW Patch Set Update : 12.1.0.2.161018 (23854735)"
   Created on 29 Sep 2016, 23:50:00 hrs PST8PDT
   Bugs fixed:
     21339083, 18589889, 20768643, 19061429, 19133945, 19341538, 19521169
     20011424, 19361757, 19187207, 19302350, 19130141, 19530755, 19699720
     19620918, 19168690, 18899171, 19244316, 19653795, 18634372, 19471722
     19536337, 19184188, 19131709, 20160490, 20995001, 21131037, 19380733
     20985167, 18943696, 20552947, 19591984, 18135723, 18461246, 20014326
     19524857, 18890943, 19154753, 19499021, 19046190, 20923930, 20028666
     19051385, 19050688, 19682695, 19831611, 21607972, 18922918, 21087799
     18956780, 19683886, 23757020, 19026993, 20110156, 21269977, 18261648
     18362971, 19558324, 22651583, 20185476, 18882642, 19478119, 19414274
     17447588, 19262534, 20752167, 19147513, 22475834, 21242118, 19514085
     20758997, 19659885, 19319904, 18703978, 23079844, 19700256, 19703246
     20467806, 19192901, 21527118, 19479503, 19147751, 18901356, 21232394
     18964974, 22162062, 18835366, 19013789, 19207286, 18950232, 22583325
     19148791, 19556820, 16505171, 22261732, 22135419, 19449737, 21878613
     21646169, 19187515, 19230771, 20795241, 19453778, 19853036, 18843572
     18520351, 21430567, 19185148, 19232454, 21220597, 18541110, 18834955
     19204743, 20175174, 19140891, 18747335, 20676340, 19270660, 19325701
     20877003, 19222693, 19662663, 19684131, 19010177, 19272663, 18910576
     18907170, 19426384, 9372680, 20737462, 19700294, 20289828, 19331454
     22568500, 20278216, 20827053, 20119857, 19703199, 22456393, 20588164
     18955644, 20991879, 18508710, 19908346, 19589221, 18952857, 19537762
     21152052, 16286734, 20669114, 18762843, 18945249, 19184799, 19205086
     19351363, 20091753, 19031737, 19079087, 19148367, 18968981, 19150517
     20231741, 19217019, 18730096, 18975620, 19067804, 21259359, 19054979
     19209951, 19318983, 18747382, 19154673, 18752378, 22905714, 19150088
     19013444, 19234177, 18999857, 19075747, 19367276, 19612597, 19874047
     19677813, 23049292, 19427050, 18910443, 20033787, 19148982, 19777496
     20730170, 18523468, 19277814, 20562133, 19071526, 21420753, 18850051
     20433932, 20061168, 19370739, 20822591, 19609388, 18776786, 20136892
     18989446, 19148793, 19585454, 18317489, 19955755, 19059053, 19807548
     19124972, 22229116, 18910748, 19147509, 18849896, 19617592, 19398098
     22840085, 19076165, 19498411, 19704993, 21294273, 19459023, 20634511
     21229744, 21474380, 21484367, 20276459, 17208793, 19234907, 19538714
     22867050, 19383028, 19513969, 22195657, 23111368, 19986391, 19314048
     18834934, 19473851, 19458082, 18894342, 19522067, 18116578, 19241857
     19076778, 19066699, 19068003, 18937186, 20267877, 19916737, 19770543
     22863466, 19635215, 19613533, 21057239, 20783416, 19315522, 19141785
     19163887, 19885321, 18715868, 18852058, 19701617, 19280860, 19139608
     21453435, 18946768, 19135521, 18604143, 19627476, 20568884, 20317221
     20177779, 19266658, 24522717, 18330979, 19027351, 21227730, 18707416
     21118157, 21113068, 19925992, 20235486, 20006646, 15922206, 18439295
     18991776, 21245437, 23170839, 19163425, 19550195, 18849021, 18861196
     17940721, 18524834, 19287706, 19522313, 18835283, 18748932, 19184765
     21639478, 14851828, 20598625, 19226141, 19053891, 18871287, 18998228
     18980002, 19682778, 19325604, 18777835, 17338864, 18945906, 19513650
     19702758, 18952577, 19473088, 19178517, 18729269, 19529729, 19455563
     20877072, 23275930, 20340620, 18536826, 19292605, 22264951, 20920711
     20660273, 20011635, 17592037, 18935857, 22989069, 19029647, 19179158
     22302421, 23076471, 19140712, 20012766, 23049301, 19184276, 20510208
     20001507, 20079414, 19680763, 19259765, 18962892, 19513888, 14661880
     21519340, 21222147, 20218012, 19551830, 21616278, 19068333, 18945435
     20775015, 19319192, 20839581, 19178629, 21694237, 19304104, 19284357
     22594310, 20725084, 19457575, 21297657, 19069755, 19021575, 18715884
     19584688, 18798573, 19812592, 19018001, 19292272, 19270956, 21967791
     21794779, 23055224, 18700893, 21276514, 20467377, 18406774, 20607821
     21976167, 21082737, 21694632, 20115586, 20883009, 19164099, 18283792
     19155350, 19884033, 19146822, 18798432, 21123424, 19471836, 20936562
     18750781, 19045143, 22483804, 19146980, 19601003, 20099624, 21378016
     19537547, 19281106, 18862203, 19954372, 18235274, 19205617, 19513351
     20219458, 19150313, 18843054, 18708349, 19332386, 18953639, 21423055
     19371270, 19203996, 20038431, 19370846, 21053036, 18998379, 20157569
     21357522, 19273577, 24709351, 22113170, 19632437, 18990354, 19288396
     20620581, 19557558, 19127078, 21792881, 20900588, 20053557, 22088198
     22664609, 19315567, 19444471, 19154933, 20953798, 18290252, 18813323
     19915945, 19500293, 18643483, 19134098, 23034352, 18965694, 19226858
     19591356, 19602208, 22884308, 20069197, 18417590, 18920408, 18636884
     22289350, 19043795, 20746507, 18260170, 18919682, 18678829, 19319357
     19273758, 18953878, 18999195, 18759724, 19066844, 13843841, 20290467
     22934855, 19649640, 19062675, 21924910, 18859710, 20862812, 19504641
     23101611, 19341481, 20293730, 19343245, 18304090, 19241655, 19470791
     18242738, 20649573, 18372060, 18953889, 18827679, 19140711, 19023430
     19259290, 19045388, 20186278, 19522571, 19709663, 18875012, 18861564
     19273760, 19225265, 15885689, 20853464, 21696068, 18819158, 21394146
     19049721, 19368917, 21255373, 22393909, 20408163, 20579351, 18868829
     20350915, 21327402, 22024217, 23101338, 19820247, 20363708, 19538241
     19804032

Patch  21436941     : applied on Thu Feb 20 17:05:29 IST 2020
Unique Patch ID:  19245012
Patch description:  "WLM Patch Set Update: 12.1.0.2.5 (21436941)"
   Created on 13 Aug 2015, 04:00:40 hrs PST8PDT
   Bugs fixed:
     20075154, 20641027, 20548410, 19016964, 19582630



--------------------------------------------------------------------------------

OPatch succeeded.
grid@suse2:/u01/app/12.1.0/grid/OPatch> pwd
/u01/app/12.1.0/grid/OPatch

grid@suse2:/u01/21231953> ll
total 168
drwxrwxr-x 5 grid oinstall   4096 Aug 29  2016 21231953
-rwxrwxr-x 1 grid oinstall 159001 Aug 29  2016 README.html
-rwxrwxr-x 1 grid oinstall      0 Aug 29  2016 README.txt
drwxrwxr-x 2 grid oinstall   4096 Aug 29  2016 automation
-rwxrwxr-x 1 grid oinstall   1572 Aug 29  2016 bundle.xml

grid@suse2:/u01/21231953> su root
Password:
suse2:/u01/21231953 # export ORACLE_HOME=/u01/app/12.1.0/grid
suse2:/u01/21231953 # export PATH=$ORACLE_HOME/OPatch:$PATH:$ORACLE_HOME/bin
suse2:/u01/21231953 # export PATH=$ORACLE_HOME/perl/bin:$PATH
suse2:/u01/21231953 # opatchauto apply -analyze

OPatchauto session is initiated at Wed Apr  8 20:37:13 2020

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-04-08_08-37-20PM.log.

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2020-04-08_08-44-14PM.log
The id for this session is RB4D

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
Patch applicability verified successfully on home /u01/app/12.1.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:suse2
CRS Home:/u01/app/12.1.0/grid
Version:12.1.0.2.0


==Following patches were SKIPPED:

Patch: /u01/21231953/21231953
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-08_20-44-28PM_1.log
Reason: /u01/21231953/21231953 is not required to be applied to oracle home /u01/app/12.1.0/grid


Host:suse2
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Version:12.1.0.2.0


==Following patches were SKIPPED:

Patch: /u01/21231953/21231953
Reason: This patch is not applicable to this specified target type - "rac_database"



OPatchauto session completed at Wed Apr  8 20:44:33 2020
Time taken to complete the session 7 minutes, 21 seconds
suse2:/u01/21231953 #
suse2:/u01/21231953 #
suse2:/u01/21231953 #
suse2:/u01/21231953 #
suse2:/u01/21231953 # opatchauto apply

OPatchauto session is initiated at Thu Apr  9 12:41:40 2020

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-04-09_12-41-43PM.log.

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2020-04-09_12-47-47PM.log
The id for this session is W9EB

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/dbhome_1

Patch applicability verified successfully on home /u01/app/12.1.0/grid


Verifying SQL patch applicability on home /u01/app/oracle/product/12.1.0/dbhome_1
No step execution required.........


Preparing to bring down database service on home /u01/app/oracle/product/12.1.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/12.1.0/dbhome_1 to bring down database service


Preparing home /u01/app/oracle/product/12.1.0/dbhome_1 after database service restarted
No step execution required.........
Doc ID 1369107.1
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:suse2
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Version:12.1.0.2.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/21231953/21231953
Reason: This patch is not applicable to this specified target type - "rac_database"


Host:suse2
CRS Home:/u01/app/12.1.0/grid
Version:12.1.0.2.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/21231953/21231953
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-09_12-48-06PM_1.log
Reason: /u01/21231953/21231953 is not required to be applied to oracle home /u01/app/12.1.0/grid



OPatchauto session completed at Thu Apr  9 12:48:14 2020
Time taken to complete the session 6 minutes, 34 seconds
suse2:/u01/21231953 #
suse2:/u01/21231953 #
suse2:/u01/21231953 #



suse1:/u01/21231953/21231953 # su grid


grid@suse1:/u01/21231953/21231953> /u01/app/12.1.0/grid/OPatch/opatch apply -oh /u01/app/12.1.0/grid -local /u01/21231953/21231953
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.1.0.2.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2020-04-09_16-23-01PM_1.log

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Patch : 21231953

        Bug SubSet of 24007012
        Subset bugs are:
        18915417, 18934139, 18899600, 19001684, 19919907, 19184398, 18900953, 20140148, 18955907, 20314907, 19557156, 19649858, 19060056, 21519796, 20681968, 19502657, 19051391, 21233961, 18957085, 18877486, 19078259, 19201087, 19610001, 19353057, 19690653, 19901825, 20098064, 19013966, 21231953, 21369858, 19526234, 20438706, 19859183, 19509898, 19053182, 19355146, 19183802, 20010980, 19528981, 19450090, 19279106, 19526283, 19644505, 19127216, 18321597, 19517835, 21555038, 19446719, 19195735, 18851012, 20402287, 19469788, 19610022, 19134464, 18185024, 19335268, 21208140, 19475588, 19452723, 19149476, 19428756, 19811784, 19270227, 18951113, 20923224, 19593769, 18510745, 21124596, 19844362

OPatch found that the following patch(es) are not required.
They are either subset of the patches in Oracle Home (or) subset of the patches in the given list:
 [ 21231953 ]

Log file location: /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2020-04-09_16-23-01PM_1.log

OPatch succeeded.
grid@suse1:/u01/21231953/21231953>

ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.12.14-120-default SP5