Understanding CMM reconfigurations

- install cluster SW on node 1 :

scinstall -i \\ 
           -C cluster32 \\ 
           -F \\ 
           -T node=c-220ra-1-epar03,node=c-220ra-2-epar03,authtype=sys \\ 
           -w netaddr=172.16.0.0,netmask=255.255.248.0,maxnodes=64,maxprivatenets=10 \\ 
           -A trtype=dlpi,name=qfe3 -A trtype=dlpi,name=qfe7 \\ 
           -B type=direct


node reboot at end of install 

node 1 messages :

ct 21 15:56:38 c-220ra-1-epar03 genunix: [ID 540533 kern.notice] \^MSunOS Release 5.10 Version Generic_120011-14 64-bit
...
Oct 21 15:56:38 c-220ra-1-epar03 swapgeneric: [ID 308332 kern.info] root on /pci@1f,4000/scsi@3/disk@1,0:a fstype ufs
...
Oct 21 15:56:40 c-220ra-1-epar03 unix: [ID 950921 kern.info] cpu0: UltraSPARC-II (portid 0 impl 0x11 ver 0xa0 clock 450 MHz)
Oct 21 15:56:40 c-220ra-1-epar03 unix: [ID 950921 kern.info] cpu2: UltraSPARC-II (portid 2 impl 0x11 ver 0xa0 clock 450 MHz)
...
Oct 21 15:57:17 c-220ra-1-epar03 pseudo: [ID 129642 kern.info] pseudo-device: clprivnet0
Oct 21 15:57:17 c-220ra-1-epar03 genunix: [ID 936769 kern.info] clprivnet0 is /pseudo/clprivnet@0
Oct 21 15:58:18 c-220ra-1-epar03 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03 (nodeid = 1) with votecount = 1 added.
Oct 21 15:58:18 c-220ra-1-epar03 cl_runtime: [ID 843983 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03: attempting to join cluster.
Oct 21 15:58:19 c-220ra-1-epar03 cl_runtime: [ID 525628 kern.notice] NOTICE: CMM: Cluster has reached quorum.
Oct 21 15:58:19 c-220ra-1-epar03 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03 (nodeid = 1) is up; new incarnation number = 1224597498.
Oct 21 15:58:19 c-220ra-1-epar03 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: c-220ra-1-epar03.
Oct 21 15:58:19 c-220ra-1-epar03 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #1 completed.
Oct 21 15:58:19 c-220ra-1-epar03 cl_runtime: [ID 499756 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03: joined cluster.
Oct 21 15:58:19 c-220ra-1-epar03 ip: [ID 856290 kern.notice] ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast
Oct 21 15:58:48 c-220ra-1-epar03 Cluster.devices.did: [ID 255440 daemon.notice] did instance 1 created.
Oct 21 15:58:48 c-220ra-1-epar03 Cluster.devices.did: [ID 224223 daemon.notice] did subpath c-220ra-1-epar03:/dev/rdsk/c0t0d0 created for instance 1.
...
Oct 21 15:58:49 c-220ra-1-epar03 Cluster.devices.did: [ID 192276 daemon.notice] did instance 26 created.
Oct 21 15:58:49 c-220ra-1-epar03 Cluster.devices.did: [ID 669180 daemon.notice] did subpath c-220ra-1-epar03:/dev/rdsk/c2t12d0 created for instance 26.
Oct 21 15:58:49 c-220ra-1-epar03 Cluster.devices.did: [ID 716564 daemon.notice] did instance 27 created.
Oct 21 15:58:49 c-220ra-1-epar03 Cluster.devices.did: [ID 144893 daemon.notice] did subpath c-220ra-1-epar03:/dev/rdsk/c2t13d0 created for instance 27.
Oct 21 15:59:16 c-220ra-1-epar03 Cluster.CCR: [ID 914260 daemon.warning] Failed to retrieve global fencing status from the global name server
Oct 21 15:59:16 c-220ra-1-epar03 Cluster.CCR: [ID 485680 daemon.warning] reservation warning(node_join) - Unable to lookup local_only flag for device dsk/d27.
...
Oct 21 15:59:16 c-220ra-1-epar03 Cluster.CCR: [ID 485680 daemon.warning] reservation warning(node_join) - Unable to lookup local_only flag for device dsk/d1.

Oct 21 16:00:14 c-220ra-1-epar03 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d1s0 is set to MONITORED
...
Oct 21 16:00:14 c-220ra-1-epar03 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d27s0 has changed to OK
Oct 21 16:00:23 c-220ra-1-epar03 Cluster.RGM.rgmd: [ID 525628 daemon.notice] CMM: Cluster has reached quorum.
Oct 21 16:00:23 c-220ra-1-epar03 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node c-220ra-1-epar03 (nodeid = 1) is up; new incarnation number = 1224597622.




cmm debug buffer :

 \*cmm_dbg_buf/s
0x3000389a008:  th 30001b9f020 tm  17323648: device_type_registry_impl::initialize()
th 30001b9f020 tm  17323695: Registered device_type_registry in nameserver
th 30001b9f020 tm  17323696: quorum_algorithm_init called
th 30001b9f020 tm  17323703: quorum_impl: Attempting to get ref to type registry
th 30001b9f020 tm  17323705: device_type_registry retrieved from nameserver.
th 30001b9f020 tm  17323706: In initialize_quorum_info
th 30001b9f020 tm  17323708: Creating event objects for quorum
th 30001b9f020 tm  17323731: initialize_device_quorum_info being called
th 30001b9f020 tm  17323732: Deleting old quorum info
th 30001b9f020 tm  17323733: 
----------Quorum Info Table-----------
th 30001b9f020 tm  17323734: List of nodes:
th 30001b9f020 tm  17323735:    Node 1: votes = 1, key = 0x48fddf3f00000001
th 30001b9f020 tm  17323737: List of quorum devices:
th 30001b9f020 tm  17323738: Total number of configured votes = 1.
th 30001b9f020 tm  17323739: 
--------End of Quorum Info Table--------

th 30001b9f020 tm  17323740: Calling process device quorum info
th 30001b9f020 tm  17323741: Returned from process device quorum info
th 30001b9f020 tm  17323742: Binding quorum to nameserver
th 30001b9f020 tm  17323744: 'quorum' bound to local name server
th 30001b9f020 tm  17329634: in unknown state
th 3000382a000 tm  17329856: CMM sender_thread starting
th 30003960ca0 tm  17329892: in start state
th 30003960340 tm  17329924: boot_delay thread starting
th 30003960ca0 tm  17329926: in check_node_health
th 30003960ca0 tm  17329928: in begin state
th 30003960ca0 tm  17329929: seqnum = 1
th 30003960ca0 tm  17329973: in connectivity state
th 30003960ca0 tm  17329974: connectivity state: membership = 0x1
th 30003960ca0 tm  17329975: Connectivity matrix while in connectivity_state:
th 30003960ca0 tm  17329976:    heard_from[1] : 0x1
th 30003960ca0 tm  17330025: in quorum_acquisition state
th 30003960ca0 tm  17330029: in delay_in_case_of_split_brain
th 30003960ca0 tm  17330030: acquire_quorum_devices(0x1) called.
th 30003960ca0 tm  17330032: partition(0x1) can have quorum
th 30003960ca0 tm  17330033: update_membership(0x1) called.
th 30003960ca0 tm  17330034: reset_quorum_devices_bus(0x1) called.
th 30003960ca0 tm  17330036: acquire_quorum_devices_lock_held returing {nodeid = 1, owned_devices = 0x0, preempted = 0}
th 30003960ca0 tm  17330045: in quorum_check state
th 30003960ca0 tm  17330050: check_cluster_quorum(): nodes=0x1, devices=0x0, votes=1, total configured votes = 1, required quorum = 1
th 30003960ca0 tm  17330052: cluster has reached quorum
th 30003960ca0 tm  17330078: in step1 state
th 30003960ca0 tm  17330384: in step2 state
th 30003960ca0 tm  17330418: in step3 state
th 30003960ca0 tm  17330447: in step4 state
th 30003960ca0 tm  17330466: in step5 state
th 30003960ca0 tm  17330576: in step6 state
th 30003960ca0 tm  17330601: in step7 state
th 30003960ca0 tm  17330764: in step8 state
th 30003960ca0 tm  17330793: in step9 state
th 30003960ca0 tm  17330814: in step10 state
th 30003960ca0 tm  17330848: in step11 state
th 30003960ca0 tm  17330877: in end_sync state
th 30003960ca0 tm  17330888: end state: seq_num = 1, membership = 0x1
th 300039adc60 tm  17332022: check_ccr_quorum(): nodes=0x0, devices=0x0, votes=0, total configured votes = 1; required quorum = 1
th 30001b9f020 tm  17332106: check_ccr_quorum(): nodes=0x1, devices=0x0, votes=1, total configured votes = 1; required quorum = 1
th 30001b9f020 tm  17346573: quorum_table_updated called
th 30001b9f020 tm  17346576: quorum_check_transition called; proposed config:
th 30001b9f020 tm  17346577: node 1: vote = 1
th 30001b9f020 tm  17346579: quorum_check_transition returning 0
th 30001b9f020 tm  17346580: No changes to quorum config
th 30001b9f020 tm  17346585: ccr_refreshed called
th 30001b9f020 tm  17346590: check_ccr_quorum(): nodes=0x1, devices=0x0, votes=1, total configured votes = 1; required quorum = 1
th 30003960340 tm  23329025: boot_delay thread exiting



- install cluster SW node 2 :


scinstall -i \\ 
           -C cluster32 \\ 
           -N c-220ra-1-epar03 \\ 
           -A trtype=dlpi,name=qfe3 -A trtype=dlpi,name=qfe7 \\ 
           -B type=direct \\ 
           -m endpoint=:qfe3,endpoint=c-220ra-1-epar03:qfe3 \\ 
           -m endpoint=:qfe7,endpoint=c-220ra-1-epar03:qfe7
           
           


messages on node 1 :
c-220ra-1-epar03# Oct 21 16:09:21 c-220ra-1-epar03 cl_runtime: NOTICE: CMM: Node c-220ra-2-epar03 (nodeid = 2) with votecount = 0 added.
Oct 21 16:09:21 c-220ra-1-epar03 cl_runtime: NOTICE: CMM: Cluster members: c-220ra-1-epar03.
Oct 21 16:09:21 c-220ra-1-epar03 cl_runtime: NOTICE: CMM: node reconfiguration #2 completed.
Oct 21 16:09:24 c-220ra-1-epar03 cl_runtime: NOTICE: clcomm: Adapter qfe3 constructed
Oct 21 16:09:25 c-220ra-1-epar03 ip: ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast
Oct 21 16:09:25 c-220ra-1-epar03 cl_runtime: NOTICE: clcomm: Adapter qfe7 constructed



cmm on node 1 :

> \*cmm_dbg_buf/s

----------Quorum Info Table-----------
th 300045bc080 tm  83626146: List of nodes:
th 300045bc080 tm  83626147:    Node 1: votes = 1, key = 0x48fddf3f00000001
th 300045bc080 tm  83626148:    Node 2: votes = 0, key = 0x48fddf3f00000002
th 300045bc080 tm  83626150: List of quorum devices:
th 300045bc080 tm  83626151: Total number of configured votes = 1.
th 300045bc080 tm  83626152: 
--------End of Quorum Info Table--------

th 300045bc080 tm  83626153: Calling process device quorum info
th 300045bc080 tm  83626154: Returned from process device quorum info
th 300045bc080 tm  83626155: acquire_quorum_devices_lock_held returing {nodeid = 1, owned_devices = 0x0, preempted = 0}
th 30003960ca0 tm  83628625: in return state
th 30003960ca0 tm  83628648: in check_node_health
th 30003960ca0 tm  83628649: in begin state
th 30003960ca0 tm  83628650: seqnum = 2
th 30003960ca0 tm  83628703: in connectivity state
th 30003960ca0 tm  83628704: connectivity state: membership = 0x1
th 30003960ca0 tm  83628705: Connectivity matrix while in connectivity_state:
th 30003960ca0 tm  83628706:    heard_from[1] : 0x1
th 30003960ca0 tm  83628721: in quorum_acquisition state
th 30003960ca0 tm  83628725: in delay_in_case_of_split_brain
th 30003960ca0 tm  83628727: acquire_quorum_devices(0x1) called.
th 30003960ca0 tm  83628729: partition(0x1) can have quorum
th 30003960ca0 tm  83628730: update_membership(0x1) called.
th 30003960ca0 tm  83628731: reset_quorum_devices_bus(0x1) called.
th 30003960ca0 tm  83628732: acquire_quorum_devices_lock_held returing {nodeid = 1, owned_devices = 0x0, preempted = 0}
th 30003960ca0 tm  83628742: in quorum_check state
th 30003960ca0 tm  83628747: check_cluster_quorum(): nodes=0x1, devices=0x0, votes=1, total configured votes = 1, required quorum = 1
th 30003960ca0 tm  83628759: in step1 state
th 30003960ca0 tm  83629074: in step2 state
th 30003960ca0 tm  83629109: in step3 state
th 30003960ca0 tm  83629142: in step4 state
th 30003960ca0 tm  83629162: in step5 state
th 30003960ca0 tm  83629188: in step6 state
th 30003960ca0 tm  83629215: in step7 state
th 30003960ca0 tm  83629243: in step8 state
th 30003960ca0 tm  83629269: in step9 state
th 30003960ca0 tm  83629298: in step10 state
th 30003960ca0 tm  83629318: in step11 state
th 30003960ca0 tm  83629350: in end_sync state
th 30003960ca0 tm  83629362: end state: seq_num = 2, membership = 0x1
th 300045bc080 tm  83721809: quorum_check_transition called; proposed config:
th 300045bc080 tm  83721810: node 1: vote = 1
th 300045bc080 tm  83721811: node 2: vote = 0
th 300045bc080 tm  83721813: quorum_check_transition returning 0
th 300045bc080 tm  83728376: quorum_table_updated called
th 300045bc080 tm  83728381: quorum_check_transition called; proposed config:
th 300045bc080 tm  83728382: node 1: vote = 1
th 300045bc080 tm  83728383: node 2: vote = 0
th 300045bc080 tm  83728384: quorum_check_transition returning 0
th 300045bc080 tm  83728385: No changes to quorum config
th 300045bc080 tm  83828088: quorum_check_transition called; proposed config:
th 300045bc080 tm  83828090: node 1: vote = 1
th 300045bc080 tm  83828091: node 2: vote = 0
th 300045bc080 tm  83828092: quorum_check_transition returning 0
th 300045bc080 tm  83835236: quorum_table_updated called
th 300045bc080 tm  83835240: quorum_check_transition called; proposed config:
th 300045bc080 tm  83835242: node 1: vote = 1
th 300045bc080 tm  83835243: node 2: vote = 0
th 300045bc080 tm  83835244: quorum_check_transition returning 0
th 300045bc080 tm  83835245: No changes to quorum config
th 300045bc080 tm  83896601: quorum_check_transition called; proposed config:
th 300045bc080 tm  83896603: node 1: vote = 1
th 300045bc080 tm  83896604: node 2: vote = 0
th 300045bc080 tm  83896605: quorum_check_transition returning 0
th 300045bc080 tm  83933187: quorum_table_updated called
th 300045bc080 tm  83933192: quorum_check_transition called; proposed config:
th 300045bc080 tm  83933193: node 1: vote = 1
th 300045bc080 tm  83933194: node 2: vote = 0
th 300045bc080 tm  83933196: quorum_check_transition returning 0
th 300045bc080 tm  83933197: No changes to quorum config
th 300045bc080 tm  84004726: quorum_check_transition called; proposed config:
th 300045bc080 tm  84004727: node 1: vote = 1
th 300045bc080 tm  84004728: node 2: vote = 0
th 300045bc080 tm  84004730: quorum_check_transition returning 0
th 300045bc080 tm  84017261: quorum_table_updated called
th 300045bc080 tm  84017266: quorum_check_transition called; proposed config:
th 300045bc080 tm  84017267: node 1: vote = 1
th 300045bc080 tm  84017268: node 2: vote = 0
th 300045bc080 tm  84017269: quorum_check_transition returning 0
th 300045bc080 tm  84017270: No changes to quorum config





node 2 reboots ,  when finished booting :

messages on node 1 :

Oct 21 16:15:15 c-220ra-1-epar03 cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node c-220ra-2-epar03 (nodeid: 2, incarnation #: 1224598511) has become reachable.
Oct 21 16:15:15 c-220ra-1-epar03 cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path c-220ra-1-epar03:qfe7 - c-220ra-2-epar03:qfe7 online
Oct 21 16:15:25 c-220ra-1-epar03 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node c-220ra-2-epar03 (nodeid = 2) is up; new incarnation number = 1224598511.
Oct 21 16:15:25 c-220ra-1-epar03 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: c-220ra-1-epar03 c-220ra-2-epar03.
Oct 21 16:15:25 c-220ra-1-epar03 Cluster.Framework: [ID 801593 daemon.notice] stdout: releasing reservations for scsi-2 disks shared with c-220ra-2-epar03
Oct 21 16:15:25 c-220ra-1-epar03 Cluster.CCR: [ID 914260 daemon.warning] Failed to retrieve global fencing status from the global name server
Oct 21 16:15:25 c-220ra-1-epar03 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #3 completed.
Oct 21 16:15:25 c-220ra-1-epar03 cl_runtime: [ID 896275 kern.notice] NOTICE: CCR: Ignoring override field for table directory on joining node c-220ra-2-epar03.
Oct 21 16:15:25 c-220ra-1-epar03 cl_runtime: [ID 896275 kern.notice] NOTICE: CCR: Ignoring override field for table dcs_service_classes on joining node c-220ra-2-epar03.
Oct 21 16:16:26 c-220ra-1-epar03 Cluster.Framework: [ID 801593 daemon.notice] stdout: releasing reservations for scsi-2 disks shared with c-220ra-2-epar03
Oct 21 16:16:26 c-220ra-1-epar03 Cluster.CCR: [ID 914260 daemon.warning] Failed to retrieve global fencing status from the global name server
...
Oct 21 16:17:34 c-220ra-1-epar03 Cluster.RGM.rgmd: [ID 537175 daemon.notice] CMM: Node c-220ra-2-epar03 (nodeid: 2, incarnation #: 1224598653) has become reachable.
Oct 21 16:17:36 c-220ra-1-epar03 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node c-220ra-2-epar03 (nodeid = 2) is up; new incarnation number = 1224598653.
Oct 21 16:17:46 c-220ra-1-epar03 Cluster.devices.did: [ID 466922 daemon.notice] obtaining access to all attached disks
Oct 21 16:17:46 c-220ra-1-epar03 Cluster.CCR: [ID 914260 daemon.warning] Failed to retrieve global fencing status from the global name server
g
c-220ra-1-epar03# 

messages on node 2 :
 done
Oct 21 16:13:31 c-220ra-2-epar03 genunix: [ID 540533 kern.notice] \^MSunOS Release 5.10 Version Generic_120011-14 64-bit
Oct 21 16:13:31 c-220ra-2-epar03 genunix: [ID 943907 kern.notice] Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
...
Oct 21 16:15:11 c-220ra-2-epar03 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03 (nodeid = 1) with votecount = 1 added.
Oct 21 16:15:11 c-220ra-2-epar03 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node c-220ra-2-epar03 (nodeid = 2) with votecount = 0 added.
Oct 21 16:15:12 c-220ra-2-epar03 cl_runtime: [ID 884114 kern.notice] NOTICE: clcomm: Adapter qfe7 constructed
Oct 21 16:15:12 c-220ra-2-epar03 cl_runtime: [ID 884114 kern.notice] NOTICE: clcomm: Adapter qfe3 constructed
Oct 21 16:15:12 c-220ra-2-epar03 cl_runtime: [ID 843983 kern.notice] NOTICE: CMM: Node c-220ra-2-epar03: attempting to join cluster.
Oct 21 16:15:14 c-220ra-2-epar03 qfe: [ID 517869 kern.info] SUNW,qfe7: 100 Mbps half duplex link up - internal  transceiver
Oct 21 16:15:15 c-220ra-2-epar03 cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03 (nodeid: 1, incarnation #: 1224597498) has become reachable.
Oct 21 16:15:15 c-220ra-2-epar03 cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path c-220ra-2-epar03:qfe7 - c-220ra-1-epar03:qfe7 online
Oct 21 16:15:24 c-220ra-2-epar03 cl_runtime: [ID 525628 kern.notice] NOTICE: CMM: Cluster has reached quorum.
Oct 21 16:15:24 c-220ra-2-epar03 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node c-220ra-1-epar03 (nodeid = 1) is up; new incarnation number = 1224597498.
Oct 21 16:15:24 c-220ra-2-epar03 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node c-220ra-2-epar03 (nodeid = 2) is up; new incarnation number = 1224598511.
Oct 21 16:15:24 c-220ra-2-epar03 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: c-220ra-1-epar03 c-220ra-2-epar03.
Oct 21 16:15:25 c-220ra-2-epar03 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #3 completed.
Oct 21 16:15:25 c-220ra-2-epar03 cl_runtime: [ID 499756 kern.notice] NOTICE: CMM: Node c-220ra-2-epar03: joined cluster.
Oct 21 16:15:26 c-220ra-2-epar03 ip: [ID 856290 kern.notice] ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast
Oct 21 16:15:57 c-220ra-2-epar03 Cluster.devices.did: [ID 183642 daemon.notice] DID subpath "/dev/rdsk/c1t0d0s2" created for instance "4".
...
"/dev/rdsk/c2t13d0s2" created for instance "27".
Oct 21 16:15:57 c-220ra-2-epar03 Cluster.devices.did: [ID 340871 daemon.notice] did instance 28 created.
Oct 21 16:15:57 c-220ra-2-epar03 Cluster.devices.did: [ID 363044 daemon.notice] did subpath c-220ra-2-epar03:/dev/rdsk/c0t0d0 created for instance 28.
Oct 21 16:15:57 c-220ra-2-epar03 Cluster.devices.did: [ID 865159 daemon.notice] did instance 29 created.
Oct 21 16:15:57 c-220ra-2-epar03 Cluster.devices.did: [ID 738738 daemon.notice] did subpath c-220ra-2-epar03:/dev/rdsk/c0t1d0 created for instance 29.
Oct 21 16:15:57 c-220ra-2-epar03 Cluster.devices.did: [ID 662856 daemon.notice] did instance 30 created.
Oct 21 16:15:58 c-220ra-2-epar03 Cluster.devices.did: [ID 623781 daemon.notice] did subpath c-220ra-2-epar03:/dev/rdsk/c0t6d0 created for instance 30.
Oct 21 16:16:12 c-220ra-2-epar03 cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path c-220ra-2-epar03:qfe3 - c-220ra-1-epar03:qfe3 errors during initiation
Oct 21 16:16:25 c-220ra-2-epar03 Cluster.CCR: [ID 914260 daemon.warning] Failed to retrieve global fencing status from the global name server
Oct 21 16:16:25 c-220ra-2-epar03 Cluster.CCR: [ID 485680 daemon.warning] reservation warning(node_join) - Unable to lookup local_only flag for device dsk/d29.

...
Oct 21 16:17:26 c-220ra-2-epar03 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d27s0 has changed to OK
Oct 21 16:17:35 c-220ra-2-epar03 Cluster.RGM.rgmd: [ID 537175 daemon.notice] CMM: Node c-220ra-1-epar03 (nodeid: 1, incarnation #: 1224597622) has become reachable.
Oct 21 16:17:36 c-220ra-2-epar03 Cluster.RGM.rgmd: [ID 525628 daemon.notice] CMM: Cluster has reached quorum.
Oct 21 16:17:36 c-220ra-2-epar03 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node c-220ra-1-epar03 (nodeid = 1) is up; new incarnation number = 1224597622.
Oct 21 16:17:36 c-220ra-2-epar03 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node c-220ra-2-epar03 (nodeid = 2) is up; new incarnation number = 1224598653.




cmm on node 1 :


> \*cmm_dbg_buf/s


----------Quorum Info Table-----------
th 300045bc080 tm  83626146: List of nodes:
th 300045bc080 tm  83626147:    Node 1: votes = 1, key = 0x48fddf3f00000001
th 300045bc080 tm  83626148:    Node 2: votes = 0, key = 0x48fddf3f00000002
th 300045bc080 tm  83626150: List of quorum devices:
th 300045bc080 tm  83626151: Total number of configured votes = 1.
th 300045bc080 tm  83626152: 
--------End of Quorum Info Table--------

th 30003960980 tm 118987154: node_is_reachable(2,1224598511); current incn = 0
th 30003960980 tm 118987157: node 2 is up; new incn = 1224598511
th 30003960ca0 tm 118987183: in return state
th 30003960ca0 tm 118987210: in check_node_health
th 30003960ca0 tm 118987211: in begin state
th 30003960ca0 tm 118987212: seqnum = 3
th 30003960ca0 tm 119468928: in connectivity state
th 30003960ca0 tm 119468945: connectivity state: membership = 0x3
th 30003960ca0 tm 119468946: Connectivity matrix while in connectivity_state:
th 30003960ca0 tm 119468947:    heard_from[1] : 0x3
th 30003960ca0 tm 119468949:    heard_from[2] : 0x3
th 30003960ca0 tm 119468979: in quorum_acquisition state
th 30003960ca0 tm 119469138: in delay_in_case_of_split_brain
th 30003960ca0 tm 119469142: acquire_quorum_devices(0x3) called.
th 30003960ca0 tm 119469143: partition(0x3) can have quorum
th 30003960ca0 tm 119469145: update_membership(0x3) called.
th 30003960ca0 tm 119469146: reset_quorum_devices_bus(0x3) called.
th 30003960ca0 tm 119469147: acquire_quorum_devices_lock_held returing {nodeid = 1, owned_devices = 0x0, preempted = 0}
th 30003960ca0 tm 119469158: in quorum_check state
th 30003960ca0 tm 119931911: check_cluster_quorum(): nodes=0x3, devices=0x0, votes=1, total configured votes = 1, required quorum = 1
th 30003960ca0 tm 119931949: in step1 state
th 30003960ca0 tm 119932545: in step2 state
th 30003960ca0 tm 119932810: in step3 state
th 30003960ca0 tm 119932948: in step4 state
th 30003960ca0 tm 119933089: in step5 state
th 30003960ca0 tm 119982318: in step6 state
th 30003960ca0 tm 119982408: in step7 state
th 30003960ca0 tm 119982761: in step8 state
th 30003960ca0 tm 119982926: in step9 state
th 30003960ca0 tm 119995994: in step10 state
th 30003960ca0 tm 119996038: in step11 state
th 30003960ca0 tm 119996285: in end_sync state
th 30003960ca0 tm 119996413: end state: seq_num = 3, membership = 0x3
>           
cmm on node 2 :

> \*cmm_dbg_buf/s

----------Quorum Info Table-----------
th 300019e4340 tm  32781805: List of nodes:
th 300019e4340 tm  32781806:    Node 1: votes = 1, key = 0x48fddf3f00000001
th 300019e4340 tm  32781807:    Node 2: votes = 0, key = 0x48fddf3f00000002
th 300019e4340 tm  32781809: List of quorum devices:
th 300019e4340 tm  32781810: Total number of configured votes = 1.
th 300019e4340 tm  32781811: 
--------End of Quorum Info Table--------

th 300019e4340 tm  32781812: Calling process device quorum info
th 300019e4340 tm  32781813: Returned from process device quorum info
th 300019e4340 tm  32781814: Binding quorum to nameserver
th 300019e4340 tm  32781816: 'quorum' bound to local name server
th 300019e4340 tm  32820958: in unknown state
th 30003950ca0 tm  32821126: CMM sender_thread starting
th 3000390b940 tm  32821147: in start state
th 3000390afe0 tm  32821168: boot_delay thread starting
th 3000390b940 tm  32821174: in check_node_health
th 3000390b940 tm  32821176: in begin state
th 3000390b940 tm  32821177: seqnum = 1
th 3000390b620 tm  33141341: node_is_reachable(1,1224597498); current incn = 0
th 3000390b620 tm  33141343: node 1 is up; new incn = 1224597498
th 300038292c0 tm  33141445: state_changed as sender seqnum greater than mine. Sender = 1
th 3000390b940 tm  33141450: seqnum = 2
th 300038292c0 tm  33141507: state_changed as sender seqnum greater than mine. Sender = 1
th 3000390b940 tm  33141513: seqnum = 3
th 3000390b940 tm  33141692: in connectivity state
th 3000390b940 tm  33141693: connectivity state: membership = 0x3
th 3000390b940 tm  33141694: Connectivity matrix while in connectivity_state:
th 3000390b940 tm  33141695:    heard_from[1] : 0x3
th 3000390b940 tm  33141696:    heard_from[2] : 0x3
th 3000390b940 tm  33623205: in quorum_acquisition state
th 3000390b940 tm  33623249: in delay_in_case_of_split_brain
th 3000390b940 tm  33623252: acquire_quorum_devices(0x3) called.
th 3000390b940 tm  33623254: partition(0x3) can have quorum
th 3000390b940 tm  33623255: update_membership(0x3) called.
th 3000390b940 tm  33623256: reset_quorum_devices_bus(0x3) called.
th 3000390b940 tm  33623257: acquire_quorum_devices_lock_held returing {nodeid = 2, owned_devices = 0x0, preempted = 0}
th 3000390b940 tm  33623268: in quorum_check state
th 3000390b940 tm  34086135: check_cluster_quorum(): nodes=0x3, devices=0x0, votes=1, total configured votes = 1, required quorum = 1
th 3000390b940 tm  34086137: cluster has reached quorum
th 3000390b940 tm  34086188: in step1 state
th 3000390b940 tm  34086799: in step2 state
th 3000390b940 tm  34087021: in step3 state
th 3000390b940 tm  34087173: in step4 state
th 3000390b940 tm  34087314: in step5 state
th 3000390b940 tm  34136434: in step6 state
th 3000390b940 tm  34136835: in step7 state
th 3000390b940 tm  34136980: in step8 state
th 3000390b940 tm  34137132: in step9 state
th 3000390b940 tm  34137270: in step10 state
th 3000390b940 tm  34150335: in step11 state
th 3000390b940 tm  34150501: in end_sync state
th 3000390b940 tm  34150631: end state: seq_num = 3, membership = 0x3
th 30003814d00 tm  34216980: ccr_refreshed called
th 3000390afe0 tm  38821023: boot_delay thread exiting

on node 1 :

# /usr/cluster/bin/scconf -c -A name=qfe3,node=1,dlpi_heartbeat_timeout=5000
# /usr/cluster/bin/scconf -c -A name=qfe3,node=1,dlpi_heartbeat_quantum=500
# /usr/cluster/bin/scconf -c -A name=qfe7,node=1,dlpi_heartbeat_timeout=5000
# /usr/cluster/bin/scconf -c -A name=qfe7,node=1,dlpi_heartbeat_quantum=500




# cat /etc/cluster/ccr/infrastructure
ccr_gennum      9
ccr_checksum    109F873F673430EFACFA4D0F6CAB0638
cluster.name    cluster32
cluster.state   enabled
cluster.properties.cluster_id   0x48FDDF3F
cluster.properties.installmode  enabled
cluster.properties.private_net_number   172.16.0.0
cluster.properties.private_netmask      255.255.248.0
cluster.properties.private_subnet_netmask       255.255.255.128
cluster.properties.private_user_net_number      172.16.4.0
cluster.properties.private_user_netmask 255.255.254.0
cluster.properties.private_maxnodes     64
cluster.properties.private_maxprivnets  10
cluster.properties.auth_joinlist_type   sys
cluster.properties.auth_joinlist_hostslist      c-220ra-1-epar03,c-220ra-2-epar03
cluster.properties.transport_heartbeat_timeout  10000
cluster.properties.transport_heartbeat_quantum  1000
cluster.properties.udp_session_timeout  480
cluster.nodes.1.name    c-220ra-1-epar03
cluster.nodes.1.state   enabled
cluster.nodes.1.properties.private_hostname     clusternode1-priv
cluster.nodes.1.properties.quorum_vote  1
cluster.nodes.1.properties.quorum_resv_key      0x48FDDF3F00000001
cluster.nodes.1.adapters.1.name qfe3
cluster.nodes.1.adapters.1.state        enabled
cluster.nodes.1.adapters.1.properties.device_name       qfe
cluster.nodes.1.adapters.1.properties.device_instance   3
cluster.nodes.1.adapters.1.properties.transport_type    dlpi
cluster.nodes.1.adapters.1.properties.lazy_free 1
cluster.nodes.1.adapters.1.properties.dlpi_heartbeat_timeout    5000
cluster.nodes.1.adapters.1.properties.dlpi_heartbeat_quantum    500
cluster.nodes.1.adapters.1.properties.nw_bandwidth      80
cluster.nodes.1.adapters.1.properties.bandwidth 10
cluster.nodes.1.adapters.1.properties.ip_address        172.16.0.129
cluster.nodes.1.adapters.1.properties.netmask   255.255.255.128
cluster.nodes.1.adapters.1.ports.1.name 0
cluster.nodes.1.adapters.1.ports.1.state        enabled
cluster.nodes.1.adapters.2.name qfe7
cluster.nodes.1.adapters.2.state        enabled
cluster.nodes.1.adapters.2.properties.device_name       qfe
cluster.nodes.1.adapters.2.properties.device_instance   7
cluster.nodes.1.adapters.2.properties.transport_type    dlpi
cluster.nodes.1.adapters.2.properties.lazy_free 1
cluster.nodes.1.adapters.2.properties.dlpi_heartbeat_timeout    5000
cluster.nodes.1.adapters.2.properties.dlpi_heartbeat_quantum    500
cluster.nodes.1.adapters.2.properties.nw_bandwidth      80
cluster.nodes.1.adapters.2.properties.bandwidth 10
cluster.nodes.1.adapters.2.properties.ip_address        172.16.1.1
cluster.nodes.1.adapters.2.properties.netmask   255.255.255.128
cluster.nodes.1.adapters.2.ports.1.name 0
cluster.nodes.1.adapters.2.ports.1.state        enabled
cluster.nodes.2.name    c-220ra-2-epar03
cluster.nodes.2.state   enabled
cluster.nodes.2.properties.quorum_vote  0
cluster.nodes.2.properties.quorum_resv_key      0x48FDDF3F00000002
cluster.nodes.2.properties.private_hostname     clusternode2-priv
cluster.nodes.2.adapters.1.name qfe3
cluster.nodes.2.adapters.1.properties.device_name       qfe
cluster.nodes.2.adapters.1.properties.device_instance   3
cluster.nodes.2.adapters.1.properties.transport_type    dlpi
cluster.nodes.2.adapters.1.properties.lazy_free 1
cluster.nodes.2.adapters.1.properties.dlpi_heartbeat_timeout    10000
cluster.nodes.2.adapters.1.properties.dlpi_heartbeat_quantum    1000
cluster.nodes.2.adapters.1.properties.nw_bandwidth      80
cluster.nodes.2.adapters.1.properties.bandwidth 10
cluster.nodes.2.adapters.1.properties.ip_address        172.16.0.130
cluster.nodes.2.adapters.1.properties.netmask   255.255.255.128
cluster.nodes.2.adapters.1.state        enabled
cluster.nodes.2.adapters.1.ports.1.name 0
cluster.nodes.2.adapters.1.ports.1.state        enabled
cluster.nodes.2.adapters.2.name qfe7
cluster.nodes.2.adapters.2.properties.device_name       qfe
cluster.nodes.2.adapters.2.properties.device_instance   7
cluster.nodes.2.adapters.2.properties.transport_type    dlpi
cluster.nodes.2.adapters.2.properties.lazy_free 1
cluster.nodes.2.adapters.2.properties.dlpi_heartbeat_timeout    10000
cluster.nodes.2.adapters.2.properties.dlpi_heartbeat_quantum    1000
cluster.nodes.2.adapters.2.properties.nw_bandwidth      80
cluster.nodes.2.adapters.2.properties.bandwidth 10
cluster.nodes.2.adapters.2.properties.ip_address        172.16.1.2
cluster.nodes.2.adapters.2.properties.netmask   255.255.255.128
cluster.nodes.2.adapters.2.state        enabled
cluster.nodes.2.adapters.2.ports.1.name 0
cluster.nodes.2.adapters.2.ports.1.state        enabled
cluster.cables.1.properties.end1        cluster.nodes.2.adapters.1.ports.1
cluster.cables.1.properties.end2        cluster.nodes.1.adapters.1.ports.1
cluster.cables.1.state  enabled
cluster.cables.2.properties.end1        cluster.nodes.2.adapters.2.ports.1
cluster.cables.2.properties.end2        cluster.nodes.1.adapters.2.ports.1
cluster.cables.2.state  enabled

on node 2 :

#/usr/cluster/bin/cluster set -p global_fencing=prefer3

=== Cluster ===                                

Cluster Name:                                   cluster32
  installmode:                                     enabled
  heartbeat_timeout:                               10000
  heartbeat_quantum:                               1000
  private_netaddr:                                 172.16.0.0
  private_netmask:                                 255.255.248.0
  max_nodes:                                       64
  max_privatenets:                                 10
  global_fencing:                                  prefer3
  Node List:                                       c-220ra-1-epar03, c-220ra-2-epar03

messages on node 1 :
Oct 21 16:25:51 c-220ra-1-epar03 Cluster.Framework: [ID 801593 daemon.notice] stdout: obtaining access to all attached disks
Oct 21 16:25:51 c-220ra-1-epar03 Cluster.CCR: [ID 429549 daemon.warning] reservation warning(node_join) - MHIOCGRP_REGISTERANDIGNOREKEY  error(48) will retry in 2 seconds
Oct 21 16:25:57 c-220ra-1-epar03 Cluster.CCR: [ID 906838 daemon.warning] reservation warning(node_join) - do_scsi3_registerandignorekey() error for disk /dev/did/rdsk/d16s2, attempting do_scsi3_register()
...
Oct 21 16:25:58 c-220ra-1-epar03 Cluster.CCR: [ID 906838 daemon.warning] reservation warning(node_join) - do_scsi3_registerandignorekey() error for disk /dev/did/rdsk/d11s2, attempting do_scsi3_register()



messages on node 2 :


Oct 21 16:25:50 c-220ra-2-epar03 Cluster.Framework: [ID 801593 daemon.notice] stdout: obtaining access to all attached disks
Oct 21 16:25:58 c-220ra-2-epar03 Cluster.CCR: [ID 429549 daemon.warning] reservation warning(node_join) - MHIOCGRP_REGISTERANDIGNOREKEY  error(48) will retry in 2 seconds
Oct 21 16:26:04 c-220ra-2-epar03 Cluster.CCR: [ID 906838 daemon.warning] reservation warning(node_join) - do_scsi3_registerandignorekey() error for disk /dev/did/rdsk/d20s2, attempting do_scsi3_register()
...
warning(node_join) - do_scsi3_registerandignorekey() error for disk /dev/did/rdsk/d12s2, attempting do_scsi3_register()


cmm node 1 :

...
th 30003960ca0 tm 119996413: end state: seq_num = 3, membership = 0x3
th 30007971020 tm 161000200: quorum_check_transition called; proposed config:
th 30007971020 tm 161000202: node 1: vote = 1
th 30007971020 tm 161000203: node 2: vote = 0
th 30007971020 tm 161000205: quorum_check_transition returning 0
th 30007971020 tm 161012538: quorum_table_updated called
th 30007971020 tm 161012542: quorum_check_transition called; proposed config:
th 30007971020 tm 161012543: node 1: vote = 1
th 30007971020 tm 161012544: node 2: vote = 0
th 30007971020 tm 161012546: quorum_check_transition returning 0
th 30007971020 tm 161012547: No changes to quorum config
th 30007971020 tm 161510875: quorum_check_transition called; proposed config:
th 30007971020 tm 161510876: node 1: vote = 1
th 30007971020 tm 161510877: node 2: vote = 0
th 30007971020 tm 161510879: quorum_check_transition returning 0
th 30007971020 tm 161524739: quorum_table_updated called
th 30007971020 tm 161524743: quorum_check_transition called; proposed config:
th 30007971020 tm 161524744: node 1: vote = 1
th 30007971020 tm 161524746: node 2: vote = 0
th 30007971020 tm 161524747: quorum_check_transition returning 0
th 30007971020 tm 161524748: No changes to quorum config
th 30007971020 tm 161986926: quorum_check_transition called; proposed config:
th 30007971020 tm 161986928: node 1: vote = 1
th 30007971020 tm 161986929: node 2: vote = 0
th 30007971020 tm 161986930: quorum_check_transition returning 0
th 30007971020 tm 162000231: quorum_table_updated called
th 30007971020 tm 162000236: quorum_check_transition called; proposed config:
th 30007971020 tm 162000237: node 1: vote = 1
th 30007971020 tm 162000238: node 2: vote = 0
th 30007971020 tm 162000239: quorum_check_transition returning 0
th 30007971020 tm 162000241: No changes to quorum config
th 30007971020 tm 162510217: quorum_check_transition called; proposed config:
th 30007971020 tm 162510218: node 1: vote = 1
th 30007971020 tm 162510219: node 2: vote = 0
th 30007971020 tm 162510221: quorum_check_transition returning 0
th 30007971020 tm 162522945: quorum_table_updated called
th 30007971020 tm 162522949: quorum_check_transition called; proposed config:
th 30007971020 tm 162522951: node 1: vote = 1
th 30007971020 tm 162522952: node 2: vote = 0
th 30007971020 tm 162522953: quorum_check_transition returning 0
th 30007971020 tm 162522955: No changes to quorum config



cmm node 2 :

> \*cmm_dbg_buf/s

th 300038149e0 tm  75169895: quorum_table_updated called
th 300038149e0 tm  75169900: quorum_check_transition called; proposed config:
th 300038149e0 tm  75169901: node 1: vote = 1
th 300038149e0 tm  75169902: node 2: vote = 0
th 300038149e0 tm  75169904: quorum_check_transition returning 0
th 300038149e0 tm  75169905: No changes to quorum config
th 300038149e0 tm  75682170: quorum_table_updated called
th 300038149e0 tm  75682174: quorum_check_transition called; proposed config:
th 300038149e0 tm  75682176: node 1: vote = 1
th 300038149e0 tm  75682177: node 2: vote = 0
th 300038149e0 tm  75682178: quorum_check_transition returning 0
th 300038149e0 tm  75682180: No changes to quorum config
th 300038149e0 tm  76157450: quorum_table_updated called
th 300038149e0 tm  76157455: quorum_check_transition called; proposed config:
th 300038149e0 tm  76157456: node 1: vote = 1
th 300038149e0 tm  76157457: node 2: vote = 0
th 300038149e0 tm  76157459: quorum_check_transition returning 0
th 300038149e0 tm  76157460: No changes to quorum config
th 300038149e0 tm  76680569: quorum_table_updated called
th 300038149e0 tm  76680573: quorum_check_transition called; proposed config:
th 300038149e0 tm  76680574: node 1: vote = 1
th 300038149e0 tm  76680575: node 2: vote = 0
th 300038149e0 tm  76680577: quorum_check_transition returning 0
th 300038149e0 tm  76680578: No changes to quorum config
>


/usr/cluster/bin/scconf -a -q globaldev=d10


messages node1 :
 21 16:31:24 c-220ra-1-epar03 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: c-220ra-1-epar03 c-220ra-2-epar03.
Oct 21 16:31:24 c-220ra-1-epar03 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #4 completed.


messages node 2 :


Oct 21 16:31:24 c-220ra-2-epar03 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: c-220ra-1-epar03 c-220ra-2-epar03.
Oct 21 16:31:24 c-220ra-2-epar03 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #4 completed.

cmm node 1 :


th 30003817020 tm 215626496: get_quorum_device called for type scsi3
th 30003817020 tm 215626497: get_quorum_device: loading module clq_scsi3
th 30003817020 tm 215630056: SCSI3: module loading.
th 30003817020 tm 215630057: device_type_registry_impl::register_device_type() register type scsi3
th 30003817020 tm 215630059: clq_scsi3: Returning new quorum device type.
th 30003817020 tm 215630133: SCSI3(/dev/did/rdsk/d10s2)::quorum_open.
th 30003817020 tm 215630308: quorum_scrub(30007b47280) called.
th 30003817020 tm 215710153: quorum_disk_initialize_reserved(30007b47280) called.
th 30003817020 tm 215712286: quorum_disk_initialize_reserved(30007b47280) retd sblkno (0x10ddbe5).
th 30003817020 tm 215712288: SCSI3(/dev/did/rdsk/d10s2) quorum_open returning QD_SUCCESS.
th 30003817020 tm 215712365: SCSI3(/dev/did/rdsk/d10s2) quorum_read_reservations called
th 30003817020 tm 215712781: SCSI3: quorum_read_reservations returned:
th 30003817020 tm 215712885: SCSI3(/dev/did/rdsk/d10s2): quorum_read_keys
th 30003817020 tm 215713317: SCSI3(/dev/did/rdsk/d10s2): quorum_read_keys returned 2 keys
th 30003817020 tm 215713320: SCSI3(/dev/did/rdsk/d10s2): quorum_read_keys returned:
th 30003817020 tm 215713321: key[0]: key=0x48fddf3f00000001
th 30003817020 tm 215713322: key[1]: key=0x48fddf3f00000002
th 30001b52ca0 tm 215765105: quorum_device_scsi3_impl::_unreferenced.
th 30003817020 tm 215775362: quorum_table_updated called
th 30003817020 tm 215775369: quorum_check_transition called; proposed config:
th 30003817020 tm 215775370: node 1: vote = 1
th 30003817020 tm 215775371: node 2: vote = 0
th 30003817020 tm 215775373: QD 1: vote = 0, name = /dev/did/rdsk/d10s2, nodes_with_configured_paths = 0x3
th 30003817020 tm 215775375: Adding QD 1
th 30003817020 tm 215775376: quorum_check_transition returning 1
th 30003817020 tm 215775377: In initialize_quorum_info
th 30003817020 tm 215775378: Creating event objects for quorum
th 30003817020 tm 215775381: initialize_device_quorum_info being called
th 30003817020 tm 215775382: Deleting old quorum info
th 30003817020 tm 215775384: 
----------Quorum Info Table-----------
th 30003817020 tm 215775385: List of nodes:
th 30003817020 tm 215775386:    Node 1: votes = 1, key = 0x48fddf3f00000001
th 30003817020 tm 215775388:    Node 2: votes = 0, key = 0x48fddf3f00000002
th 30003817020 tm 215775389: List of quorum devices:
th 30003817020 tm 215775390:    Quorum device 1: gdevname = '/dev/did/rdsk/d10s2', votes = 0, nodes_with_configured_paths = 0x3
th 30003817020 tm 215775391: Total number of configured votes = 1.
th 30003817020 tm 215775393: 
--------End of Quorum Info Table--------

th 30003817020 tm 215775394: Calling process device quorum info
th 30003817020 tm 215775395: Returned from process device quorum info
th 30003817020 tm 215775396: acquire_quorum_devices_lock_held returing {nodeid = 1, owned_devices = 0x0, preempted = 0}
th 30003960ca0 tm 215823208: in return state
th 30003960ca0 tm 215823244: in check_node_health
th 30003960ca0 tm 215823245: in begin state
th 30003960ca0 tm 215823246: seqnum = 4
th 30003960ca0 tm 215823569: in connectivity state
th 30003960ca0 tm 215823570: connectivity state: membership = 0x3
th 30003960ca0 tm 215823571: Connectivity matrix while in connectivity_state:
th 30003960ca0 tm 215823572:    heard_from[1] : 0x3
th 30003960ca0 tm 215823574:    heard_from[2] : 0x3
th 30003960ca0 tm 215823588: in quorum_acquisition state
th 30003960ca0 tm 215824906: in delay_in_case_of_split_brain
th 30003960ca0 tm 215824909: acquire_quorum_devices(0x3) called.
th 30003960ca0 tm 215824910: partition(0x3) can have quorum
th 30003960ca0 tm 215824912: update_membership(0x3) called.
th 30003960ca0 tm 215824913: reset_quorum_devices_bus(0x3) called.
th 30003960ca0 tm 215824915: acquire_quorum_devices_lock_held returing {nodeid = 1, owned_devices = 0x0, preempted = 0}
th 30003960ca0 tm 215824926: in quorum_check state
th 30003960ca0 tm 215824952: check_cluster_quorum(): nodes=0x3, devices=0x0, votes=1, total configured votes = 1, required quorum = 1
th 30003960ca0 tm 215824969: in step1 state
th 30003960ca0 tm 215866399: in step2 state
th 30003960ca0 tm 215866875: in step3 state
th 30003960ca0 tm 215866988: in step4 state
th 30003960ca0 tm 215867161: in step5 state
th 30003960ca0 tm 215867313: in step6 state
th 30003960ca0 tm 215867450: in step7 state
th 30003960ca0 tm 215867627: in step8 state
th 30003960ca0 tm 215867776: in step9 state
th 30003960ca0 tm 215867914: in step10 state
th 30003960ca0 tm 215868070: in step11 state
th 30003960ca0 tm 215868225: in end_sync state
th 30003960ca0 tm 215868362: end state: seq_num = 4, membership = 0x3
th 30003817020 tm 215943372: quorum_table_updated called
th 30003817020 tm 215943379: quorum_check_transition called; proposed config:
th 30003817020 tm 215943380: node 1: vote = 1
th 30003817020 tm 215943381: node 2: vote = 0
th 30003817020 tm 215943383: QD 1: vote = 0, name = /dev/did/rdsk/d10s2, nodes_with_configured_paths = 0x3
th 30003817020 tm 215943385: quorum_check_transition returning 0
th 30003817020 tm 215943386: No changes to quorum config






cmm node 2 :

> \*cmm_dbg_buf/s

th 30005bb4080 tm 129874267: quorum_check_transition called; proposed config:
th 30005bb4080 tm 129874269: node 1: vote = 1
th 30005bb4080 tm 129874270: node 2: vote = 0
th 30005bb4080 tm 129874271: QD 1: vote = 0, name = /dev/did/rdsk/d10s2, nodes_with_configured_paths = 0x3            
th 30005bb4080 tm 129874273: Adding QD 1
th 30005bb4080 tm 129874274: quorum_check_transition returning 1
th 300038149e0 tm 129932606: quorum_table_updated called
th 300038149e0 tm 129932612: quorum_check_transition called; proposed config:
th 300038149e0 tm 129932614: node 1: vote = 1
th 300038149e0 tm 129932615: node 2: vote = 0
th 300038149e0 tm 129932616: QD 1: vote = 0, name = /dev/did/rdsk/d10s2, nodes_with_configured_paths = 0x3
th 300038149e0 tm 129932617: Adding QD 1
th 300038149e0 tm 129932618: quorum_check_transition returning 1
th 300038149e0 tm 129932620: In initialize_quorum_info
th 300038149e0 tm 129932621: Creating event objects for quorum
th 300038149e0 tm 129932624: initialize_device_quorum_info being called
th 300038149e0 tm 129932625: Deleting old quorum info
th 300038149e0 tm 129932627: 
----------Quorum Info Table-----------
th 300038149e0 tm 129932628: List of nodes:
th 300038149e0 tm 129932629:    Node 1: votes = 1, key = 0x48fddf3f00000001
th 300038149e0 tm 129932630:    Node 2: votes = 0, key = 0x48fddf3f00000002
th 300038149e0 tm 129932632: List of quorum devices:
th 300038149e0 tm 129932633:    Quorum device 1: gdevname = '/dev/did/rdsk/d10s2', votes = 0, nodes_with_configured_paths = 0x3
th 300038149e0 tm 129932634: Total number of configured votes = 1.
th 300038149e0 tm 129932635: 
--------End of Quorum Info Table--------

th 300038149e0 tm 129932636: Calling process device quorum info
th 300038149e0 tm 129932638: Returned from process device quorum info
th 300038149e0 tm 129932639: acquire_quorum_devices_lock_held returing {nodeid = 2, owned_devices = 0x0, preempted = 0}
th 3000390b940 tm 129976103: in return state
th 3000390b940 tm 129976130: in check_node_health
th 3000390b940 tm 129976132: in begin state
th 3000390b940 tm 129976133: seqnum = 4
th 3000390b940 tm 129976437: in connectivity state
th 3000390b940 tm 129976438: connectivity state: membership = 0x3
th 3000390b940 tm 129976441: Connectivity matrix while in connectivity_state:
th 3000390b940 tm 129976442:    heard_from[1] : 0x3
th 3000390b940 tm 129976443:    heard_from[2] : 0x3
th 3000390b940 tm 129976636: in quorum_acquisition state
th 3000390b940 tm 129976655: in delay_in_case_of_split_brain
th 3000390b940 tm 129976659: acquire_quorum_devices(0x3) called.
th 3000390b940 tm 129976660: partition(0x3) can have quorum
th 3000390b940 tm 129976662: update_membership(0x3) called.
th 3000390b940 tm 129976663: reset_quorum_devices_bus(0x3) called.
th 3000390b940 tm 129976664: acquire_quorum_devices_lock_held returing {nodeid = 2, owned_devices = 0x0, preempted = 0}
th 3000390b940 tm 129976673: in quorum_check state
th 3000390b940 tm 129977983: check_cluster_quorum(): nodes=0x3, devices=0x0, votes=1, total configured votes = 1, required quorum = 1
th 3000390b940 tm 129977998: in step1 state
th 3000390b940 tm 130019431: in step2 state
th 3000390b940 tm 130019869: in step3 state
th 3000390b940 tm 130020040: in step4 state
th 3000390b940 tm 130020192: in step5 state
th 3000390b940 tm 130020346: in step6 state
th 3000390b940 tm 130020488: in step7 state
th 3000390b940 tm 130020664: in step8 state
th 3000390b940 tm 130020800: in step9 state
th 3000390b940 tm 130020952: in step10 state
th 3000390b940 tm 130021105: in step11 state
th 3000390b940 tm 130021268: in end_sync state
th 3000390b940 tm 130021393: end state: seq_num = 4, membership = 0x3
th 30005bb4080 tm 130036565: quorum_check_transition called; proposed config:
th 30005bb4080 tm 130036566: node 1: vote = 1
th 30005bb4080 tm 130036567: node 2: vote = 0
th 30005bb4080 tm 130036568: QD 1: vote = 0, name = /dev/did/rdsk/d10s2, nodes_with_configured_paths = 0x3
th 30005bb4080 tm 130036571: quorum_check_transition returning 0
th 30003815660 tm 130100495: quorum_table_updated called
th 30003815660 tm 130100501: quorum_check_transition called; proposed config:
th 30003815660 tm 130100503: node 1: vote = 1
th 30003815660 tm 130100504: node 2: v
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Jean-Christophe Lamoure

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today