RAC - CRS-2403: The Cluster Time Synchronization Service on host 02 is in observer mode.

Scenario : We have 2 node 12c 12.1.0.2.0 grid clusterware setup ,  Time synchronization service is being managed by clusterware ctssd service.

Some how our linux guys make time sync entry between two nodes with NTP Server without downtime. Due to this CTSS is gone to  Observer state. Switching over to clock synchronization checks using NTP without any downtime. My post motive was  just to share error details.


CRS alert log below:
/u01/app/grid/diag/crs/02/crs/trace/alert.log


2020-04-13 12:58:49.279 [OCTSSD(4195)]CRS-2403: The Cluster Time Synchronization Service on host 02 is in observer mode.
2020-04-13 13:00:09.296 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.
2020-04-13 13:30:10.200 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.

2020-04-13 12:58:47.221 [OCTSSD(1477)]CRS-2403: The Cluster Time Synchronization Service on host 01 is in observer mode.
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
02,01
PRVF-5415 : Check to see if NTP daemon or service is running failed

grid@01:~> . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/grid
grid@01:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

grid@01:~> ssh sgdcplm02
Last login: Tue Apr 14 09:14:25 2020 from 10.4.6.138
grid@02:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.


##############################################################################

grid@01:/u01/app/12.1.0/grid/bin> cluvfy comp clocksync -n all -verbose

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  01                             passed
  02                             passed
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  02                             Observer
  01                             Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
  Node Name                             File exists?
  ------------------------------------  ------------------------
  02                             yes
  01                             yes
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  02                             no
  01                             no
PRVF-7590 : "ntpd" is not running on node "02"
PRVF-7590 : "ntpd" is not running on node "01"
PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed


PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
grid@01:/u01/app/12.1.0/grid/bin> cat /etc/ntp.conf

############################################################################

Alert log after changing to ntp  in octssd.trc

grid@01:/u01/app/grid/diag/crs/01/crs/trace> tail -1000f octssd.trc

2020-04-14 10:27:54.403648 :    CTSS:3791980288: sclsctss_gvss3: NTP active, forcing observer mode
2020-04-14 10:27:54.403654 :    CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
2020-04-14 10:28:06.943670 :    CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xee], offset[0 ms]}, length=[8].
2020-04-14 10:28:09.554924 :GIPCHTHR:3817641728:  gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-14 10:28:19.557075 :GIPCHTHR:3815540480:  gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-14 10:28:24.406977 :    CTSS:3791980288: sclsctss_ivsr1: default config file found


old_octssd.trc

2020-04-12 00:39:02.478503 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478517 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478521 :    CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:07.378919 :    CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:19.984439 :GIPCHTHR:3817641728:  gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-12 00:39:31.991080 :GIPCHTHR:3815540480:  gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:39:32.481366 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481379 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481382 :    CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:37.383441 :    CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:50.988957 :GIPCHTHR:3817641728:  gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-12 00:40:01.992865 :GIPCHTHR:3815540480:  gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:40:02.485423 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:40:02.485450 :    CTSS:3791980288: sclsctss_ivsr2: default pid file not found

grid@02:~>crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUP.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.CRS.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.DATA.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.FRA1.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.FRA2.dg
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.asm
               ONLINE  ONLINE       01                Started,STABLE
               ONLINE  ONLINE       02                Started,STABLE
ora.net1.network
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
ora.ons
               ONLINE  ONLINE       01                STABLE
               ONLINE  ONLINE       02                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       02                STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       01                STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       01                STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       01                169.254.238.75 192.1
                                                             68.0.63,STABLE
ora.cvu
      1        ONLINE  ONLINE       01                STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       01                Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       01                STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       02                STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       01                STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       01                STABLE
ora.01.vip
      1        ONLINE  ONLINE       01                STABLE
ora.02.vip
      1        ONLINE  ONLINE       02                STABLE
ora.rac.acrac.svc
      1        ONLINE  ONLINE       01                STABLE
ora.rac.db
      1        ONLINE  ONLINE       02                Open,STABLE
      2        ONLINE  ONLINE       01                Open,STABLE
ora.rac.pretaf.svc
      1        ONLINE  ONLINE       02                STABLE
ora.rac.pretaf_preconnect.svc
      1        ONLINE  ONLINE       01                STABLE
ora.rac.staf.svc
      1        ONLINE  ONLINE       01                STABLE
--------------------------------------------------------------------------------