Scenario Preview:
We have installed 2 node grid 12c 12.1.0.2.0 setup on virtual box. It was running fine
But we got some issue on virtual machine.
While starting both the cluster one node got up and another node face issue. While checking on cluster alert log and occsd.trc log found below error as ocssd agent and ocssd monitor service unable to communicate to another node.
HAIP service was running but found issue on occsd service
As per troubleshooting issue find some issue on private IP, Both node private IP unable to communicate with each other , Once fix the issue able to up node2.
Please find error below.
occsd.trc
Checking multicast communication...
Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11138 : Interface "192.168.10.2" on node "srv2" is not able to communicate with interface "192.168.10.1" on node "srv1" over multicast group "224.0.0.251"
Checking subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0"...
PRVG-11138 : Interface "192.168.10.1" on node "srv1" is not able to communicate with interface "192.168.10.2" on node "srv2" over multicast group "230.0.1.0"
PRVG-11138 : Interface "192.168.10.2" on node "srv2" is not able to communicate with interface "192.168.10.1" on node "srv1" over multicast group "230.0.1.0"
Checking subnet "192.168.10.0" for broadcast communication with broadcast address "255.255.255.255"...
PRVG-11352 : Interface "192.168.10.1" on node "srv1" is not able to communicate with interface "192.168.10.2" on node "srv2" with broadcast address "255.255.255.255"
PRVG-11352 : Interface "192.168.10.2" on node "srv2" is not able to communicate with interface "192.168.10.1" on node "srv1" with broadcast address "255.255.255.255"
2019-08-26 18:23:23.287791 : CSSD:3585390336: clssnmvDHBValidateNCopy: node 2, srv2, has a disk HB, but no network HB, DHB has rcfg 462308150, wrtcnt, 793334, LATS 262364, lastSeqNo 793331, uniqueness 1566822324, timestamp 1566824004/1447474
2019-08-26 18:23:23.384245 : CSSD:3574351616: clssnmRcfgMgrThread: Local Join
2019-08-26 18:23:23.384251 : CSSD:3574351616: clssnmLocalJoinEvent: begin on node(1), waittime 193000
2019-08-26 18:23:23.384253 : CSSD:3574351616: clssnmLocalJoinEvent: set curtime (262464) for my node
2019-08-26 18:23:23.384254 : CSSD:3574351616: clssnmLocalJoinEvent: scanning 32 nodes
2019-08-26 18:23:23.384258 : CSSD:3574351616: clssnmLocalJoinEvent: Node srv2, number 2, is in an existing cluster with disk state 3
2019-08-26 18:23:23.384271 : CSSD:3574351616: clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2019-08-26 18:23:23.683061 : CSSD:3964958464: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2019-08-26 18:23:23.683066 : CSSD:3964958464: clsssc_CLSFAInit_CB: clsfa fencing not ready yet
2019-08-26 18:23:24.124345 : CSSD:3617572608: clssscWaitOnEventValue: after CmInfo State val 3, eval 1 waited 1000 with cvtimewait status 4294967186
2019-08-26 18:23:24.288870 : CSSD:3590121216: clssnmvDHBValidateNCopy: node 2, srv2, has a disk HB, but no network HB, DHB has rcfg 462308150, wrtcnt, 793335, LATS 263364, lastSeqNo 793332, uniqueness 1566822324, timestamp 1566824004/1448014
2019-08-26 18:23:24.313298 : CSSD:3585390336: clssnmvDHBValidateNCopy: node 2, srv2, has a disk HB, but no network HB, DHB has rcfg 462308150, wrtcnt, 793337, LATS 263394, lastSeqNo 793334, uniqueness 1566822324, timestamp 1566824005/1448484
2019-08-26 18:23:24.683818 : CSSD:3964958464: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2019-08-26 18:23:24.683823 : CSSD:3964958464: clsssc_CLSFAInit_CB: clsfa fencing not ready yet
2019-08-26 18:23:25.124376 : CSSD:3617572608: clssscWaitOnEventValue: after CmInfo State val 3, eval 1 waited 1000 with cvtimewait status 4294967186
2019-08-26 18:23:25.353656 : CSSD:3590121216: clssnmvDHBValidateNCopy: node 2, srv2, has a disk HB, but no network HB, DHB has rcfg 462308150, wrtcnt, 793338, LATS 264434, lastSeqNo 793335, uniqueness 1566822324, timestamp 1566824005/1449034
2019-08-26 18:23:25.363754 : CSSD:3585390336: clssnmvDHBValidateNCopy: node 2, srv2, has a disk HB, but no network HB, DHB has rcfg 462308150, wrtcnt, 793340, LATS 264444, lastSeqNo 793337, uniqueness 1566822324, timestamp 1566824006/1449484
2019-08-26 18:23:25.684749 : CSSD:3964958464: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2019-08-26 18:23:25.684754 : CSSD:3964958464: clsssc_CLSFAInit_CB: clsfa fencing not ready yet
NOTE : Both Private IP unable to communicate each other.
[root@srv1 ~]# ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
From 192.168.10.1 icmp_seq=1 Destination Host Unreachable
From 192.168.10.1 icmp_seq=2 Destination Host Unreachable
From 192.168.10.1 icmp_seq=3 Destination Host Unreachable
From 192.168.10.1 icmp_seq=4 Destination Host Unreachable
From 192.168.10.1 icmp_seq=5 Destination Host Unreachable
From 192.168.10.1 icmp_seq=6 Destination Host Unreachable
From 192.168.10.1 icmp_seq=7 Destination Host Unreachable
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
From 192.168.10.2 icmp_seq=1 Destination Host Unreachable
From 192.168.10.2 icmp_seq=2 Destination Host Unreachable
From 192.168.10.2 icmp_seq=3 Destination Host Unreachable
From 192.168.10.2 icmp_seq=4 Destination Host Unreachable
[root@srv1 bin]# ./cluvfy stage -pre dbinst -fixup -n nodea,nodeb -osdba dba -verbose
You must NOT be logged in as root (uid=0) when running ./cluvfy.
[root@srv1 bin]# su grid
[grid@srv1 bin]$ ./cluvfy stage -pre dbinst -fixup -n nodea,nodeb -osdba dba -verbose
Performing pre-checks for database installation
Checking node reachability...
PRVF-6006 : unable to reach the IP addresses "nodea,nodeb" from the local node
PRKN-1034 : Failed to retrieve IP address of host "nodea"
PRKN-1034 : Failed to retrieve IP address of host "nodeb"
ERROR:
Unable to reach any of the nodes
Verification cannot proceed
Pre-check for database installation was unsuccessful on all the nodes.
NOTE:
No fixable verification failures to fix
[grid@srv1 bin]$ ./cluvfy stage -pre dbinst -fixup -n srv1,srv2 -osdba dba -verbose
Performing pre-checks for database installation
Checking node reachability...
Check: Node reachability from node "srv1"
Destination Node Reachable?
------------------------------------ ------------------------
srv1 yes
srv2 yes
Result: Node reachability check passed from node "srv1"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
srv2 passed
srv1 passed
Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
srv1 passed
srv2 passed
Verification of the hosts config file successful
Interface information for node "srv1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.56.71 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.91 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.92 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.93 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.81 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.82 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth1 192.168.10.1 192.168.10.0 0.0.0.0 UNKNOWN 08:00:27:5C:15:9A 1500
eth1 169.254.157.132 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:5C:15:9A 1500
Interface information for node "srv2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.56.72 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:80:49:0B 1500
eth1 192.168.10.2 192.168.10.0 0.0.0.0 UNKNOWN 08:00:27:7E:5A:5A 1500
Check: Node connectivity using interfaces on subnet "192.168.56.0"
Check: Node connectivity of subnet "192.168.56.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv1[192.168.56.71] srv2[192.168.56.72] yes
srv1[192.168.56.71] srv1[192.168.56.81] yes
srv1[192.168.56.71] srv1[192.168.56.92] yes
srv1[192.168.56.71] srv1[192.168.56.82] yes
srv1[192.168.56.71] srv1[192.168.56.93] yes
srv1[192.168.56.71] srv1[192.168.56.91] yes
srv2[192.168.56.72] srv1[192.168.56.81] yes
srv2[192.168.56.72] srv1[192.168.56.92] yes
srv2[192.168.56.72] srv1[192.168.56.82] yes
srv2[192.168.56.72] srv1[192.168.56.93] yes
srv2[192.168.56.72] srv1[192.168.56.91] yes
srv1[192.168.56.81] srv1[192.168.56.92] yes
srv1[192.168.56.81] srv1[192.168.56.82] yes
srv1[192.168.56.81] srv1[192.168.56.93] yes
srv1[192.168.56.81] srv1[192.168.56.91] yes
srv1[192.168.56.92] srv1[192.168.56.82] yes
srv1[192.168.56.92] srv1[192.168.56.93] yes
srv1[192.168.56.92] srv1[192.168.56.91] yes
srv1[192.168.56.82] srv1[192.168.56.93] yes
srv1[192.168.56.82] srv1[192.168.56.91] yes
srv1[192.168.56.93] srv1[192.168.56.91] yes
Result: Node connectivity passed for subnet "192.168.56.0" with node(s) srv1,srv2
Check: TCP connectivity of subnet "192.168.56.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv1 : 192.168.56.71 srv1 : 192.168.56.71 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.82 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.91 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.71 srv2 : 192.168.56.72 passed
srv2 : 192.168.56.72 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.81 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.92 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.82 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.93 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.91 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.81 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.82 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.91 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.92 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.82 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.91 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.82 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.82 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.82 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.82 passed
srv1 : 192.168.56.82 srv1 : 192.168.56.82 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.82 passed
srv1 : 192.168.56.91 srv1 : 192.168.56.82 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.93 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.82 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.91 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.91 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.91 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.91 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.91 passed
srv1 : 192.168.56.82 srv1 : 192.168.56.91 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.91 passed
srv1 : 192.168.56.91 srv1 : 192.168.56.91 passed
Result: TCP connectivity check passed for subnet "192.168.56.0"
Check: Node connectivity using interfaces on subnet "192.168.10.0"
Check: Node connectivity of subnet "192.168.10.0"
WARNING:
Make sure IP address "eth1 : 192.168.10.1 [192.168.10.0] " is up and is a valid IP address on node "srv1"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv2[192.168.10.2] srv1[192.168.10.1] no
ERROR:
PRVF-7616 : Node connectivity failed for subnet "192.168.10.0" between "srv2 - eth1 : 192.168.10.2" and "srv1 - eth1 : 192.168.10.1"
Result: Node connectivity failed for subnet "192.168.10.0"
Check: TCP connectivity of subnet "192.168.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv2 : 192.168.10.2 srv2 : 192.168.10.2 passed
srv1 : 192.168.10.1 srv2 : 192.168.10.2 failed
ERROR:
PRVG-11850 : The system call "connect" failed with error "113" while executing exectask on node "srv1"
No route to host
srv2 : 192.168.10.2 srv1 : 192.168.10.1 failed
ERROR:
PRVG-11850 : The system call "connect" failed with error "113" while executing exectask on node "srv2"
No route to host
srv1 : 192.168.10.1 srv1 : 192.168.10.1 passed
Result: TCP connectivity check failed for subnet "192.168.10.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed.
Result: Node connectivity check failed
Checking multicast communication...
Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11138 : Interface "192.168.10.1" on node "srv1" is not able to communicate with interface "192.168.10.2" on node "srv2" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.10.2" on node "srv2" is not able to communicate with interface "192.168.10.1" on node "srv1" over multicast group "224.0.0.251"
Checking subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0"...
PRVG-11138 : Interface "192.168.10.1" on node "srv1" is not able to communicate with interface "192.168.10.2" on node "srv2" over multicast group "230.0.1.0"
PRVG-11138 : Interface "192.168.10.2" on node "srv2" is not able to communicate with interface "192.168.10.1" on node "srv1" over multicast group "230.0.1.0"
Checking subnet "192.168.10.0" for broadcast communication with broadcast address "255.255.255.255"...
PRVG-11352 : Interface "192.168.10.1" on node "srv1" is not able to communicate with interface "192.168.10.2" on node "srv2" with broadcast address "255.255.255.255"
PRVG-11352 : Interface "192.168.10.2" on node "srv2" is not able to communicate with interface "192.168.10.1" on node "srv1" with broadcast address "255.255.255.255"
Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 5.579GB (5849972.0KB) 1GB (1048576.0KB) passed
srv1 5.579GB (5849972.0KB) 1GB (1048576.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 5.1522GB (5402460.0KB) 50MB (51200.0KB) passed
srv1 3.4969GB (3666744.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 5.8437GB (6127612.0KB) 5.579GB (5849972.0KB) passed
srv1 5.8437GB (6127612.0KB) 5.579GB (5849972.0KB) passed
Result: Swap space check passed
Check: Free disk space for "srv2:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp srv2 / 53.4688GB 1GB passed
Result: Free disk space check passed for "srv2:/tmp"
Check: Free disk space for "srv1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp srv1 / 52.9111GB 1GB passed
Result: Free disk space check passed for "srv1:/tmp"
Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists(54323)
srv1 passed exists(54323)
Checking for multiple users with UID value 54323
Result: Check for multiple users with UID value 54323 passed
Result: User existence check passed for "grid"
Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "dba"
Check: Group existence for "asmdba"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "asmdba"
Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Status
---------------- ------------ ------------ ------------ ------------ ------------
srv2 yes yes yes yes passed
srv1 yes yes yes yes passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed
Check: Membership of user "grid" in group "dba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "dba" passed
Check: Membership of user "grid" in group "asmdba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "asmdba" passed
Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
srv2 5 3,5 passed
srv1 5 3,5 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 hard 65536 65536 passed
srv1 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 soft 1024 1024 passed
srv1 soft 65536 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 hard 16384 16384 passed
srv1 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 soft 16384 2047 passed
srv1 soft 16384 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 x86_64 x86_64 passed
srv1 x86_64 x86_64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 3.8.13-68.3.4.el6uek.x86_64 2.6.39 passed
srv1 3.8.13-68.3.4.el6uek.x86_64 2.6.39 passed
Result: Kernel version check passed
Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 250 250 250 passed
srv2 250 250 250 passed
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 32000 32000 32000 passed
srv2 32000 32000 32000 passed
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 100 100 100 passed
srv2 100 100 100 passed
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 128 128 128 passed
srv2 128 128 128 passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4398046511104 4398046511104 2995185664 passed
srv2 4398046511104 4398046511104 2995185664 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4096 4096 4096 passed
srv2 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4294967296 4294967296 584997 passed
srv2 4294967296 4294967296 584997 passed
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 6815744 6815744 6815744 passed
srv2 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
srv2 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 262144 262144 262144 passed
srv2 262144 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4194304 4194304 4194304 passed
srv2 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 262144 262144 262144 passed
srv2 262144 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 1048576 1048576 1048576 passed
srv2 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 1048576 1048576 1048576 passed
srv2 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 binutils-2.20.51.0.2-5.43.el6 binutils-2.20.51.0.2 passed
srv1 binutils-2.20.51.0.2-5.43.el6 binutils-2.20.51.0.2 passed
Result: Package existence check passed for "binutils"
Check: Package existence for "compat-libcap1"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
srv1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
Result: Package existence check passed for "compat-libcap1"
Check: Package existence for "compat-libstdc++-33(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
srv1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "libgcc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libgcc(x86_64)-4.4.7-16.el6 libgcc(x86_64)-4.4.4 passed
srv1 libgcc(x86_64)-4.4.7-16.el6 libgcc(x86_64)-4.4.4 passed
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libstdc++(x86_64)-4.4.7-16.el6 libstdc++(x86_64)-4.4.4 passed
srv1 libstdc++(x86_64)-4.4.7-16.el6 libstdc++(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libstdc++-devel(x86_64)-4.4.7-16.el6 libstdc++-devel(x86_64)-4.4.4 passed
srv1 libstdc++-devel(x86_64)-4.4.7-16.el6 libstdc++-devel(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 sysstat-9.0.4-27.el6 sysstat-9.0.4 passed
srv1 sysstat-9.0.4-27.el6 sysstat-9.0.4 passed
Result: Package existence check passed for "sysstat"
Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 gcc-4.4.7-16.el6 gcc-4.4.4 passed
srv1 gcc-4.4.7-16.el6 gcc-4.4.4 passed
Result: Package existence check passed for "gcc"
Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 gcc-c++-4.4.7-16.el6 gcc-c++-4.4.4 passed
srv1 gcc-c++-4.4.7-16.el6 gcc-c++-4.4.4 passed
Result: Package existence check passed for "gcc-c++"
Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 ksh ksh passed
srv1 ksh ksh passed
Result: Package existence check passed for "ksh"
Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 make-3.81-20.el6 make-3.81 passed
srv1 make-3.81-20.el6 make-3.81 passed
Result: Package existence check passed for "make"
Check: Package existence for "glibc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 glibc(x86_64)-2.12-1.166.el6 glibc(x86_64)-2.12 passed
srv1 glibc(x86_64)-2.12-1.166.el6 glibc(x86_64)-2.12 passed
Result: Package existence check passed for "glibc(x86_64)"
Check: Package existence for "glibc-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 glibc-devel(x86_64)-2.12-1.166.el6 glibc-devel(x86_64)-2.12 passed
srv1 glibc-devel(x86_64)-2.12-1.166.el6 glibc-devel(x86_64)-2.12 passed
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "libaio(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
srv1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "libaio-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
srv1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio-devel(x86_64)"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
srv2 passed
srv1 passed
Check for consistency of root user's primary group passed
Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
srv2 0022 0022 passed
srv1 0022 0022 passed
Result: Default user file creation mask check passed
Checking CRS integrity...
Clusterware version consistency passed.
The Oracle Clusterware is healthy on node "srv1"
ERROR:
PRVF-5305 : The Oracle Clusterware is not healthy on node "srv2"
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
CRS integrity check failed
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
------------------------------------ ------------------------
srv1 running
srv2 not running
ERROR:
PRVF-5319 : Oracle Cluster Synchronization Services do not appear to be online.
Cluster manager integrity check failed
Checking node application existence...
Checking existence of VIP node application (required)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
srv1 yes yes passed
srv2 yes no exists
VIP node application is offline on nodes "srv2"
Checking existence of NETWORK node application (required)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
srv1 yes yes passed
srv2 yes no failed
PRVF-4570 : Failed to check existence of NETWORK node application on nodes "srv2"
Checking existence of ONS node application (optional)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
srv1 no yes passed
srv2 no no failed
PRVF-4576 : Failed to check existence of ONS node application on nodes "srv2"
Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
srv1 passed
srv2 failed
PRVG-1453 : Oracle CTSS resource is not in ONLINE state on nodes "srv2"
Check of CTSS resource passed on all nodes
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
srv1 Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
Node Name Time Offset Status
------------ ------------------------ ------------------------
srv1 0.0 passed
Time offset is within the specified limits on the following set of nodes:
"[srv1]"
Result: Check of clock time offsets passed
PRVF-9652 : Cluster Time Synchronization Services check failed
Checking integrity of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of 'domain' and 'search' entries is defined
WARNING:
PRVF-5640 : Both 'search' and 'domain' entries are present in file "/etc/resolv.conf" on the following nodes: srv1,srv2
Check for integrity of file "/etc/resolv.conf" passed
Check: Time zone consistency
Result: Time zone consistency check passed
Checking Single Client Access Name (SCAN)...
SCAN Name Node Running? ListenerName Port Running?
---------------- ------------ ------------ ------------ ------------ ------------
srv-scan srv1 true LISTENER_SCAN1 1521 true
srv-scan srv1 true LISTENER_SCAN2 1521 true
srv-scan srv1 true LISTENER_SCAN3 1521 true
Checking TCP connectivity to SCAN listeners...
Node ListenerName TCP connectivity?
------------ ------------------------ ------------------------
srv1 LISTENER_SCAN1 yes
srv1 LISTENER_SCAN2 yes
srv1 LISTENER_SCAN3 yes
TCP connectivity to SCAN listeners exists on all cluster nodes
Checking name resolution setup for "srv-scan"...
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
SCAN Name IP Address Status Comment
------------ ------------------------ ------------------------ ----------
srv-scan 192.168.56.91 passed
srv-scan 192.168.56.93 passed
srv-scan 192.168.56.92 passed
Checking SCAN IP addresses...
Check of SCAN IP addresses passed
Verification of SCAN VIP and listener setup passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.
Checking Database and Clusterware version compatibility
Checking ASM and CRS version compatibility
ASM and CRS versions are compatible
Database version "12.1" is compatible with the Clusterware version "12.1.0.2.0".
Database Clusterware version compatibility passed.
Check: Group existence for "asmadmin"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "asmadmin"
Check: Membership of user "grid" in group "asmadmin"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "asmadmin" passed
Check: Group existence for "asmdba"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "asmdba"
Check: Membership of user "grid" in group "asmdba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "asmdba" passed
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passed
Starting check for Maximum locked memory check ...
ERROR:
PRVE-0044 : No entry in configuration file when checking locked memory limit on node "srv2.example.com"
PRVE-0044 : No entry in configuration file when checking locked memory limit on node "srv1.example.com"
Check for Maximum locked memory check failed
Pre-check for database installation was unsuccessful on all the nodes.
******************************************************************************************
Following is the list of fixable prerequisites selected to fix in this session
******************************************************************************************
-------------- --------------- ----------------
Check failed. Failed on nodes Reboot required?
-------------- --------------- ----------------
Maximum locked memory check srv2,srv1 no
Execute "/tmp/CVU_12.1.0.2.0_grid/runfixup.sh" as root user on nodes "srv1,srv2" to perform the fix up operations manually
Press ENTER key to continue after execution of "/tmp/CVU_12.1.0.2.0_grid/runfixup.sh" has completed on nodes "srv1,srv2"
Fix: Maximum locked memory check
Node Name Status
------------------------------------ ------------------------
srv2 successful
srv1 successful
Result: "Maximum locked memory check" was successfully fixed on all the applicable nodes
Fix up operations were successfully completed on all the applicable nodes
[grid@srv1 bin]$
[root@srv2 ~]# ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=0.009 ms
64 bytes from 192.168.10.2: icmp_seq=2 ttl=64 time=0.015 ms
64 bytes from 192.168.10.2: icmp_seq=3 ttl=64 time=0.028 ms
64 bytes from 192.168.10.2: icmp_seq=4 ttl=64 time=0.015 ms
^Z
[10]+ Stopped ping 192.168.10.2
[root@srv2 ~]# ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.346 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.189 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.252 ms
^Z
[11]+ Stopped ping 192.168.10.1
[root@srv2 ~]#
[root@srv2 ~]#
[root@srv2 ~]#
[root@srv2 ~]# ps -ef |grep pmon
root 32160 32030 0 13:05 pts/0 00:00:00 grep pmon
[root@srv2 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
[root@srv2 ~]# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@srv2 ~]# crsctl start cluster
CRS-2672: Attempting to start 'ora.crf' on 'srv2'
CRS-2672: Attempting to start 'ora.cssd' on 'srv2'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv2'
CRS-2676: Start of 'ora.diskmon' on 'srv2' succeeded
CRS-2676: Start of 'ora.crf' on 'srv2' succeeded
CRS-2676: Start of 'ora.cssd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'srv2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2676: Start of 'ora.ctssd' on 'srv2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv2'
CRS-2676: Start of 'ora.asm' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'srv2'
CRS-2676: Start of 'ora.storage' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'srv2'
CRS-2676: Start of 'ora.crsd' on 'srv2' succeeded
[root@srv2 ~]#
###########################################################################
[grid@srv1 bin]$ ./cluvfy stage -pre dbinst -fixup -n srv1,srv2 -osdba dba -verbose
Performing pre-checks for database installation
Checking node reachability...
Check: Node reachability from node "srv1"
Destination Node Reachable?
------------------------------------ ------------------------
srv1 yes
srv2 yes
Result: Node reachability check passed from node "srv1"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
srv2 passed
srv1 passed
Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
srv1 passed
srv2 passed
Verification of the hosts config file successful
Interface information for node "srv1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.56.71 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.92 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.93 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth0 192.168.56.81 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:A4:14:0A 1500
eth1 192.168.10.1 192.168.10.0 0.0.0.0 UNKNOWN 08:00:27:5C:15:9A 1500
eth1 169.254.157.132 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:5C:15:9A 1500
Interface information for node "srv2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.56.72 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:80:49:0B 1500
eth0 192.168.56.82 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:80:49:0B 1500
eth0 192.168.56.91 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:80:49:0B 1500
eth1 192.168.10.2 192.168.10.0 0.0.0.0 UNKNOWN 08:00:27:7E:5A:5A 1500
eth1 169.254.69.177 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:7E:5A:5A 1500
Check: Node connectivity using interfaces on subnet "192.168.56.0"
Check: Node connectivity of subnet "192.168.56.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv2[192.168.56.91] srv2[192.168.56.72] yes
srv2[192.168.56.91] srv1[192.168.56.93] yes
srv2[192.168.56.91] srv2[192.168.56.82] yes
srv2[192.168.56.91] srv1[192.168.56.71] yes
srv2[192.168.56.91] srv1[192.168.56.92] yes
srv2[192.168.56.91] srv1[192.168.56.81] yes
srv2[192.168.56.72] srv1[192.168.56.93] yes
srv2[192.168.56.72] srv2[192.168.56.82] yes
srv2[192.168.56.72] srv1[192.168.56.71] yes
srv2[192.168.56.72] srv1[192.168.56.92] yes
srv2[192.168.56.72] srv1[192.168.56.81] yes
srv1[192.168.56.93] srv2[192.168.56.82] yes
srv1[192.168.56.93] srv1[192.168.56.71] yes
srv1[192.168.56.93] srv1[192.168.56.92] yes
srv1[192.168.56.93] srv1[192.168.56.81] yes
srv2[192.168.56.82] srv1[192.168.56.71] yes
srv2[192.168.56.82] srv1[192.168.56.92] yes
srv2[192.168.56.82] srv1[192.168.56.81] yes
srv1[192.168.56.71] srv1[192.168.56.92] yes
srv1[192.168.56.71] srv1[192.168.56.81] yes
srv1[192.168.56.92] srv1[192.168.56.81] yes
Result: Node connectivity passed for subnet "192.168.56.0" with node(s) srv2,srv1
Check: TCP connectivity of subnet "192.168.56.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv2 : 192.168.56.91 srv2 : 192.168.56.91 passed
srv2 : 192.168.56.72 srv2 : 192.168.56.91 passed
srv1 : 192.168.56.93 srv2 : 192.168.56.91 passed
srv2 : 192.168.56.82 srv2 : 192.168.56.91 passed
srv1 : 192.168.56.71 srv2 : 192.168.56.91 passed
srv1 : 192.168.56.92 srv2 : 192.168.56.91 passed
srv1 : 192.168.56.81 srv2 : 192.168.56.91 passed
srv2 : 192.168.56.91 srv2 : 192.168.56.72 passed
srv2 : 192.168.56.72 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.93 srv2 : 192.168.56.72 passed
srv2 : 192.168.56.82 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.71 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.92 srv2 : 192.168.56.72 passed
srv1 : 192.168.56.81 srv2 : 192.168.56.72 passed
srv2 : 192.168.56.91 srv1 : 192.168.56.93 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.93 passed
srv2 : 192.168.56.82 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.93 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.93 passed
srv2 : 192.168.56.91 srv2 : 192.168.56.82 passed
srv2 : 192.168.56.72 srv2 : 192.168.56.82 passed
srv1 : 192.168.56.93 srv2 : 192.168.56.82 passed
srv2 : 192.168.56.82 srv2 : 192.168.56.82 passed
srv1 : 192.168.56.71 srv2 : 192.168.56.82 passed
srv1 : 192.168.56.92 srv2 : 192.168.56.82 passed
srv1 : 192.168.56.81 srv2 : 192.168.56.82 passed
srv2 : 192.168.56.91 srv1 : 192.168.56.71 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.71 passed
srv2 : 192.168.56.82 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.71 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.71 passed
srv2 : 192.168.56.91 srv1 : 192.168.56.92 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.92 passed
srv2 : 192.168.56.82 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.92 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.92 passed
srv2 : 192.168.56.91 srv1 : 192.168.56.81 passed
srv2 : 192.168.56.72 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.93 srv1 : 192.168.56.81 passed
srv2 : 192.168.56.82 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.71 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.92 srv1 : 192.168.56.81 passed
srv1 : 192.168.56.81 srv1 : 192.168.56.81 passed
Result: TCP connectivity check passed for subnet "192.168.56.0"
Check: Node connectivity using interfaces on subnet "192.168.10.0"
Check: Node connectivity of subnet "192.168.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv2[192.168.10.2] srv1[192.168.10.1] yes
Result: Node connectivity passed for subnet "192.168.10.0" with node(s) srv2,srv1
Check: TCP connectivity of subnet "192.168.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
srv2 : 192.168.10.2 srv2 : 192.168.10.2 passed
srv1 : 192.168.10.1 srv2 : 192.168.10.2 passed
srv2 : 192.168.10.2 srv1 : 192.168.10.1 passed
srv1 : 192.168.10.1 srv1 : 192.168.10.1 passed
Result: TCP connectivity check passed for subnet "192.168.10.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 5.579GB (5849972.0KB) 1GB (1048576.0KB) passed
srv1 5.579GB (5849972.0KB) 1GB (1048576.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 4.2886GB (4496904.0KB) 50MB (51200.0KB) passed
srv1 3.467GB (3635412.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 5.8437GB (6127612.0KB) 5.579GB (5849972.0KB) passed
srv1 5.8437GB (6127612.0KB) 5.579GB (5849972.0KB) passed
Result: Swap space check passed
Check: Free disk space for "srv2:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp srv2 / 53.4609GB 1GB passed
Result: Free disk space check passed for "srv2:/tmp"
Check: Free disk space for "srv1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp srv1 / 52.8955GB 1GB passed
Result: Free disk space check passed for "srv1:/tmp"
Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists(54323)
srv1 passed exists(54323)
Checking for multiple users with UID value 54323
Result: Check for multiple users with UID value 54323 passed
Result: User existence check passed for "grid"
Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "dba"
Check: Group existence for "asmdba"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "asmdba"
Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Status
---------------- ------------ ------------ ------------ ------------ ------------
srv2 yes yes yes yes passed
srv1 yes yes yes yes passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed
Check: Membership of user "grid" in group "dba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "dba" passed
Check: Membership of user "grid" in group "asmdba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "asmdba" passed
Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
srv2 5 3,5 passed
srv1 5 3,5 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 hard 65536 65536 passed
srv1 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 soft 1024 1024 passed
srv1 soft 65536 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 hard 16384 16384 passed
srv1 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
srv2 soft 16384 2047 passed
srv1 soft 16384 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 x86_64 x86_64 passed
srv1 x86_64 x86_64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 3.8.13-68.3.4.el6uek.x86_64 2.6.39 passed
srv1 3.8.13-68.3.4.el6uek.x86_64 2.6.39 passed
Result: Kernel version check passed
Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 250 250 250 passed
srv2 250 250 250 passed
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 32000 32000 32000 passed
srv2 32000 32000 32000 passed
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 100 100 100 passed
srv2 100 100 100 passed
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 128 128 128 passed
srv2 128 128 128 passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4398046511104 4398046511104 2995185664 passed
srv2 4398046511104 4398046511104 2995185664 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4096 4096 4096 passed
srv2 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4294967296 4294967296 584997 passed
srv2 4294967296 4294967296 584997 passed
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 6815744 6815744 6815744 passed
srv2 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
srv2 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 262144 262144 262144 passed
srv2 262144 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 4194304 4194304 4194304 passed
srv2 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 262144 262144 262144 passed
srv2 262144 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 1048576 1048576 1048576 passed
srv2 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
srv1 1048576 1048576 1048576 passed
srv2 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 binutils-2.20.51.0.2-5.43.el6 binutils-2.20.51.0.2 passed
srv1 binutils-2.20.51.0.2-5.43.el6 binutils-2.20.51.0.2 passed
Result: Package existence check passed for "binutils"
Check: Package existence for "compat-libcap1"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
srv1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
Result: Package existence check passed for "compat-libcap1"
Check: Package existence for "compat-libstdc++-33(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
srv1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "libgcc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libgcc(x86_64)-4.4.7-16.el6 libgcc(x86_64)-4.4.4 passed
srv1 libgcc(x86_64)-4.4.7-16.el6 libgcc(x86_64)-4.4.4 passed
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libstdc++(x86_64)-4.4.7-16.el6 libstdc++(x86_64)-4.4.4 passed
srv1 libstdc++(x86_64)-4.4.7-16.el6 libstdc++(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libstdc++-devel(x86_64)-4.4.7-16.el6 libstdc++-devel(x86_64)-4.4.4 passed
srv1 libstdc++-devel(x86_64)-4.4.7-16.el6 libstdc++-devel(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 sysstat-9.0.4-27.el6 sysstat-9.0.4 passed
srv1 sysstat-9.0.4-27.el6 sysstat-9.0.4 passed
Result: Package existence check passed for "sysstat"
Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 gcc-4.4.7-16.el6 gcc-4.4.4 passed
srv1 gcc-4.4.7-16.el6 gcc-4.4.4 passed
Result: Package existence check passed for "gcc"
Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 gcc-c++-4.4.7-16.el6 gcc-c++-4.4.4 passed
srv1 gcc-c++-4.4.7-16.el6 gcc-c++-4.4.4 passed
Result: Package existence check passed for "gcc-c++"
Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 ksh ksh passed
srv1 ksh ksh passed
Result: Package existence check passed for "ksh"
Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 make-3.81-20.el6 make-3.81 passed
srv1 make-3.81-20.el6 make-3.81 passed
Result: Package existence check passed for "make"
Check: Package existence for "glibc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 glibc(x86_64)-2.12-1.166.el6 glibc(x86_64)-2.12 passed
srv1 glibc(x86_64)-2.12-1.166.el6 glibc(x86_64)-2.12 passed
Result: Package existence check passed for "glibc(x86_64)"
Check: Package existence for "glibc-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 glibc-devel(x86_64)-2.12-1.166.el6 glibc-devel(x86_64)-2.12 passed
srv1 glibc-devel(x86_64)-2.12-1.166.el6 glibc-devel(x86_64)-2.12 passed
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "libaio(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
srv1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "libaio-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
srv2 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
srv1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio-devel(x86_64)"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
srv2 passed
srv1 passed
Check for consistency of root user's primary group passed
Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
srv2 0022 0022 passed
srv1 0022 0022 passed
Result: Default user file creation mask check passed
Checking CRS integrity...
Clusterware version consistency passed.
The Oracle Clusterware is healthy on node "srv1"
The Oracle Clusterware is healthy on node "srv2"
CRS integrity check passed
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
------------------------------------ ------------------------
srv1 running
srv2 running
Oracle Cluster Synchronization Services appear to be online.
Cluster manager integrity check passed
Checking node application existence...
Checking existence of VIP node application (required)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
srv1 yes yes passed
srv2 yes yes passed
VIP node application check passed
Checking existence of NETWORK node application (required)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
srv1 yes yes passed
srv2 yes yes passed
NETWORK node application check passed
Checking existence of ONS node application (optional)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
srv1 no yes passed
srv2 no yes passed
ONS node application check passed
Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
srv1 passed
srv2 passed
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
srv2 Active
srv1 Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
Node Name Time Offset Status
------------ ------------------------ ------------------------
srv2 0.0 passed
srv1 0.0 passed
Time offset is within the specified limits on the following set of nodes:
"[srv2, srv1]"
Result: Check of clock time offsets passed
Oracle Cluster Time Synchronization Services check passed
Checking integrity of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of 'domain' and 'search' entries is defined
WARNING:
PRVF-5640 : Both 'search' and 'domain' entries are present in file "/etc/resolv.conf" on the following nodes: srv1,srv2
Check for integrity of file "/etc/resolv.conf" passed
Check: Time zone consistency
Result: Time zone consistency check passed
Checking Single Client Access Name (SCAN)...
SCAN Name Node Running? ListenerName Port Running?
---------------- ------------ ------------ ------------ ------------ ------------
srv-scan srv2 true LISTENER_SCAN1 1521 true
srv-scan srv1 true LISTENER_SCAN2 1521 true
srv-scan srv1 true LISTENER_SCAN3 1521 true
Checking TCP connectivity to SCAN listeners...
Node ListenerName TCP connectivity?
------------ ------------------------ ------------------------
srv1 LISTENER_SCAN1 yes
srv1 LISTENER_SCAN2 yes
srv1 LISTENER_SCAN3 yes
TCP connectivity to SCAN listeners exists on all cluster nodes
Checking name resolution setup for "srv-scan"...
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
SCAN Name IP Address Status Comment
------------ ------------------------ ------------------------ ----------
srv-scan 192.168.56.91 passed
srv-scan 192.168.56.93 passed
srv-scan 192.168.56.92 passed
Checking SCAN IP addresses...
Check of SCAN IP addresses passed
Verification of SCAN VIP and listener setup passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.
Checking Database and Clusterware version compatibility
Checking ASM and CRS version compatibility
ASM and CRS versions are compatible
Database version "12.1" is compatible with the Clusterware version "12.1.0.2.0".
Database Clusterware version compatibility passed.
Check: Group existence for "asmadmin"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "asmadmin"
Check: Membership of user "grid" in group "asmadmin"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "asmadmin" passed
Check: Group existence for "asmdba"
Node Name Status Comment
------------ ------------------------ ------------------------
srv2 passed exists
srv1 passed exists
Result: Group existence check passed for "asmdba"
Check: Membership of user "grid" in group "asmdba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
srv2 yes yes yes passed
srv1 yes yes yes passed
Result: Membership check for user "grid" in group "asmdba" passed
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passed
Starting check for Maximum locked memory check ...
Check for Maximum locked memory check passed
Pre-check for database installation was successful.
NOTE:
No fixable verification failures to fix
[grid@srv1 bin]$
No comments:
Post a Comment