Scenario : We have 2 node 12c 12.1.0.2.0 grid clusterware setup , Time synchronization service is being managed by clusterware ctssd service.
Some how our linux guys make time sync entry between two nodes with NTP Server without downtime. Due to this CTSS is gone to Observer state. Switching over to clock synchronization checks using NTP without any downtime. My post motive was just to share error details.
CRS alert log below:
/u01/app/grid/diag/crs/02/crs/trace/alert.log
2020-04-13 12:58:49.279 [OCTSSD(4195)]CRS-2403: The Cluster Time Synchronization Service on host 02 is in observer mode.
2020-04-13 13:00:09.296 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.
2020-04-13 13:30:10.200 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.
2020-04-13 12:58:47.221 [OCTSSD(1477)]CRS-2403: The Cluster Time Synchronization Service on host 01 is in observer mode.
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
02,01
PRVF-5415 : Check to see if NTP daemon or service is running failed
grid@01:~> . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/grid
grid@01:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
grid@01:~> ssh sgdcplm02
Last login: Tue Apr 14 09:14:25 2020 from 10.4.6.138
grid@02:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
##############################################################################
grid@01:/u01/app/12.1.0/grid/bin> cluvfy comp clocksync -n all -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
01 passed
02 passed
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
02 Observer
01 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
Node Name File exists?
------------------------------------ ------------------------
02 yes
01 yes
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
02 no
01 no
PRVF-7590 : "ntpd" is not running on node "02"
PRVF-7590 : "ntpd" is not running on node "01"
PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
grid@01:/u01/app/12.1.0/grid/bin> cat /etc/ntp.conf
############################################################################
Alert log after changing to ntp in octssd.trc
grid@01:/u01/app/grid/diag/crs/01/crs/trace> tail -1000f octssd.trc
2020-04-14 10:27:54.403648 : CTSS:3791980288: sclsctss_gvss3: NTP active, forcing observer mode
2020-04-14 10:27:54.403654 : CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
2020-04-14 10:28:06.943670 : CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xee], offset[0 ms]}, length=[8].
2020-04-14 10:28:09.554924 :GIPCHTHR:3817641728: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-14 10:28:19.557075 :GIPCHTHR:3815540480: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-14 10:28:24.406977 : CTSS:3791980288: sclsctss_ivsr1: default config file found
old_octssd.trc
2020-04-12 00:39:02.478503 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478517 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478521 : CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:07.378919 : CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:19.984439 :GIPCHTHR:3817641728: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-12 00:39:31.991080 :GIPCHTHR:3815540480: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:39:32.481366 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481379 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481382 : CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:37.383441 : CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:50.988957 :GIPCHTHR:3817641728: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-12 00:40:01.992865 :GIPCHTHR:3815540480: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:40:02.485423 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:40:02.485450 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
grid@02:~>crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUP.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.CRS.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.DATA.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.FRA1.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.FRA2.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.asm
ONLINE ONLINE 01 Started,STABLE
ONLINE ONLINE 02 Started,STABLE
ora.net1.network
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.ons
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE 02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE 01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE 01 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE 01 169.254.238.75 192.1
68.0.63,STABLE
ora.cvu
1 ONLINE ONLINE 01 STABLE
ora.mgmtdb
1 ONLINE ONLINE 01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE 01 STABLE
ora.scan1.vip
1 ONLINE ONLINE 02 STABLE
ora.scan2.vip
1 ONLINE ONLINE 01 STABLE
ora.scan3.vip
1 ONLINE ONLINE 01 STABLE
ora.01.vip
1 ONLINE ONLINE 01 STABLE
ora.02.vip
1 ONLINE ONLINE 02 STABLE
ora.rac.acrac.svc
1 ONLINE ONLINE 01 STABLE
ora.rac.db
1 ONLINE ONLINE 02 Open,STABLE
2 ONLINE ONLINE 01 Open,STABLE
ora.rac.pretaf.svc
1 ONLINE ONLINE 02 STABLE
ora.rac.pretaf_preconnect.svc
1 ONLINE ONLINE 01 STABLE
ora.rac.staf.svc
1 ONLINE ONLINE 01 STABLE
--------------------------------------------------------------------------------
Some how our linux guys make time sync entry between two nodes with NTP Server without downtime. Due to this CTSS is gone to Observer state. Switching over to clock synchronization checks using NTP without any downtime. My post motive was just to share error details.
CRS alert log below:
/u01/app/grid/diag/crs/02/crs/trace/alert.log
2020-04-13 12:58:49.279 [OCTSSD(4195)]CRS-2403: The Cluster Time Synchronization Service on host 02 is in observer mode.
2020-04-13 13:00:09.296 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.
2020-04-13 13:30:10.200 [OCTSSD(4195)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/02/crs/trace/octssd.trc.
2020-04-13 12:58:47.221 [OCTSSD(1477)]CRS-2403: The Cluster Time Synchronization Service on host 01 is in observer mode.
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
02,01
PRVF-5415 : Check to see if NTP daemon or service is running failed
grid@01:~> . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/grid
grid@01:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
grid@01:~> ssh sgdcplm02
Last login: Tue Apr 14 09:14:25 2020 from 10.4.6.138
grid@02:~> crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
##############################################################################
grid@01:/u01/app/12.1.0/grid/bin> cluvfy comp clocksync -n all -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
01 passed
02 passed
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
02 Observer
01 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
Node Name File exists?
------------------------------------ ------------------------
02 yes
01 yes
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
02 no
01 no
PRVF-7590 : "ntpd" is not running on node "02"
PRVF-7590 : "ntpd" is not running on node "01"
PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
grid@01:/u01/app/12.1.0/grid/bin> cat /etc/ntp.conf
############################################################################
Alert log after changing to ntp in octssd.trc
grid@01:/u01/app/grid/diag/crs/01/crs/trace> tail -1000f octssd.trc
2020-04-14 10:27:54.403654 : CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
2020-04-14 10:28:06.943670 : CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xee], offset[0 ms]}, length=[8].
2020-04-14 10:28:09.554924 :GIPCHTHR:3817641728: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-14 10:28:19.557075 :GIPCHTHR:3815540480: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-14 10:28:24.406977 : CTSS:3791980288: sclsctss_ivsr1: default config file found
old_octssd.trc
2020-04-12 00:39:02.478503 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478517 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:02.478521 : CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:07.378919 : CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:19.984439 :GIPCHTHR:3817641728: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31010loopCount 39
2020-04-12 00:39:31.991080 :GIPCHTHR:3815540480: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:39:32.481366 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481379 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:39:32.481382 : CTSS:3791980288: ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2020-04-12 00:39:37.383441 : CTSS:3819742976: ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2020-04-12 00:39:50.988957 :GIPCHTHR:3817641728: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 31000loopCount 39
2020-04-12 00:40:01.992865 :GIPCHTHR:3815540480: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30010loopCount 37
2020-04-12 00:40:02.485423 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
2020-04-12 00:40:02.485450 : CTSS:3791980288: sclsctss_ivsr2: default pid file not found
grid@02:~>crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUP.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.CRS.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.DATA.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.FRA1.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.FRA2.dg
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.asm
ONLINE ONLINE 01 Started,STABLE
ONLINE ONLINE 02 Started,STABLE
ora.net1.network
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
ora.ons
ONLINE ONLINE 01 STABLE
ONLINE ONLINE 02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE 02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE 01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE 01 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE 01 169.254.238.75 192.1
68.0.63,STABLE
ora.cvu
1 ONLINE ONLINE 01 STABLE
ora.mgmtdb
1 ONLINE ONLINE 01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE 01 STABLE
ora.scan1.vip
1 ONLINE ONLINE 02 STABLE
ora.scan2.vip
1 ONLINE ONLINE 01 STABLE
ora.scan3.vip
1 ONLINE ONLINE 01 STABLE
ora.01.vip
1 ONLINE ONLINE 01 STABLE
ora.02.vip
1 ONLINE ONLINE 02 STABLE
ora.rac.acrac.svc
1 ONLINE ONLINE 01 STABLE
ora.rac.db
1 ONLINE ONLINE 02 Open,STABLE
2 ONLINE ONLINE 01 Open,STABLE
ora.rac.pretaf.svc
1 ONLINE ONLINE 02 STABLE
ora.rac.pretaf_preconnect.svc
1 ONLINE ONLINE 01 STABLE
ora.rac.staf.svc
1 ONLINE ONLINE 01 STABLE
--------------------------------------------------------------------------------
No comments:
Post a Comment