Scenario Preview :
we have 2 node grid 12c . After upgrade unable to start instance on node1 so after check alertlog file we found some error below after troubleshooting found doc (Doc ID 1546393.1) and increases value at OS level.
Operating System : SLES 12 SP4
we have 2 node grid 12c . After upgrade unable to start instance on node1 so after check alertlog file we found some error below after troubleshooting found doc (Doc ID 1546393.1) and increases value at OS level.
Error:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process m000 died, see its trace file
alert_SMRPLM1.log
srv1:/u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace # tail -1000f alert_SMRPLM1.log
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Sun Jul 26 20:53:23 2020
Process startup failed, error stack:
Sun Jul 26 20:53:24 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process m000 died, see its trace file
Sun Jul 26 20:53:26 2020
Process startup failed, error stack:
Sun Jul 26 20:53:26 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Process m000 died, see its trace file
Sun Jul 26 20:53:46 2020
Instance reconfiguration: current system load 19 (hiload)
Sun Jul 26 20:53:52 2020
Reconfiguration started (old inc 2, new inc 4)
List of instances (total 2) :
1 2
New instances (total 1) :
2
My inst 1
Sun Jul 26 20:54:09 2020
Global Resource Directory frozen
Starting background process RSMN
Sun Jul 26 20:54:09 2020
Process startup failed, error stack:
Sun Jul 26 20:54:09 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_psp0_6541.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Sun Jul 26 20:54:10 2020
NOTE: ASMB connected to ASM instance +ASM1 osid: 6613 (Flex mode; client id 0x10001)
Sun Jul 26 20:54:10 2020
Communication channels reestablished
Sun Jul 26 20:54:10 2020
NOTE: initiating MARK startup
Sun Jul 26 20:54:10 2020
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Process RSMN died, see its trace file
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Sun Jul 26 20:54:10 2020
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Sun Jul 26 20:54:11 2020
Errors in file /u01/app/oracle/diag/rdbms/smrplm/SMRPLM1/trace/SMRPLM1_ora_6432.trc:
ORA-00443: background process "RSMN" did not start
Sun Jul 26 20:54:11 2020
USER (ospid: 6432): terminating the instance due to error 450
Sun Jul 26 20:54:14 2020
Instance terminated by USER, pid = 6432
**********************************************************************
srv1:~ # crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.CRS.dg
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.DATA.dg
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.FRA.dg
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.net1.network
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.ons
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE srv2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE srv1 169.254.185.74 192.1
68.10.1,STABLE
ora.asm
1 ONLINE ONLINE srv1 Started,STABLE
2 ONLINE ONLINE srv2 Started,STABLE
ora.cvu
1 ONLINE ONLINE srv1 STABLE
ora.mgmtdb
1 ONLINE ONLINE srv1 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE srv2 STABLE
ora.scan1.vip
1 ONLINE ONLINE srv2 STABLE
ora.scan2.vip
1 ONLINE ONLINE srv1 STABLE
ora.scan3.vip
1 ONLINE ONLINE srv1 STABLE
ora.smrplm.db
1 ONLINE OFFLINE Instance Shutdown,ST
ABLE
2 ONLINE ONLINE srv2 Open,HOME=/u01/app/o
racle/product/12.1.0
/dbhome_1,STABLE
ora.smrplm.pretafplm.svc
1 ONLINE ONLINE srv2 STABLE
ora.smrplm.pretafplm_preconnect.svc
1 ONLINE OFFLINE STABLE
ora.srv1.vip
1 ONLINE ONLINE srv1 STABLE
ora.srv2.vip
1 ONLINE ONLINE srv2 STABLE
--------------------------------------------------------------------------------
Solution :
Following ORA-27300: OS System Dependent Operation:fork Failed With Status: 11 (Doc ID 1546393.1)
Increase the nporc value on file /etc/security/limits.conf
grid soft nproc 131072
Also check value on /etc/sysctl.conf file :
srv1:/home/oracle # sysctl -w kernel.pid_max=139264
kernel.pid_max = 139264
sysctl -p
oracle@srv1:~> sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Sun Jul 26 21:04:50 2020
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2147483648 bytes
Fixed Size 2926472 bytes
Variable Size 1140852856 bytes
Database Buffers 989855744 bytes
Redo Buffers 13848576 bytes
Database mounted.
Database opened.
SQL>
oracle@srv1:~> srvctl status database -d SMRPLM
Instance SMRPLM1 is running on node srv1
Instance SMRPLM2 is running on node srv2
Still in some case i have seen error occurred set value below at OS file
# cat /etc/systemd/system.conf set value DefaultTasksMax=65535
No comments:
Post a Comment