Optional fencing

What is Fencing ?


Fencing is a means to control updates to shared data when an active primary node fails. It does not protect against malicious users or programs. Access to shared devices controlled via scsi-2 or scsi-3 reservations.



Why allowing user to turn off fencing ?


Because not all storage out in the field supports scsi-2 or scsi-3, and qualifying a storage is a lengthy process to ensure scsi-2 and scsi-3 compliance. Sometime fencing interferes with complex storage topologies in which outside systems need access to shared cluster storage. This is new to suncluster 3.2 update 2.

New CLI to turn off fencing


Enhancements to the existing scsi-3 override and quorum configuration CLIs: scdidadm -G nofencing

scdidadm -F nofencing instance

cldevice set -p default_fencing=nofencing instance

cluster set -p global_fencing=nofencing

c-220ra-1-epar03 # cluster set -p global_fencing=nofencing
Updating shared devices on node 1
Updating shared devices on node 2

c-220ra-1-epar03 # cluster set -p global_fencing=nofencing-noscrub

c-220ra-1-epar03 # cluster set -p global_fencing=pathcount
Updating shared devices on node 1
Updating shared devices on node 2

c-220ra-1-epar03 # cluster set -p global_fencing=prefer3
Updating shared devices on node 1
Updating shared devices on node 2

c-220ra-1-epar03 # scdidadm -F nofencing d5
Updating shared devices on node 1
Updating shared devices on node 2


c-220ra-1-epar03 # scdidadm -G
The cluster is currently configured to use prefer SCSI3 algorithm to determine fencing protocol for shared devices

clquorum – changed type from scsi to shared_disk

c-220ra-1-epar03 # clquorum show

=== Cluster Nodes ===                          

Node Name:                                      c-220ra-1-epar03
  Node ID:                                         1
  Quorum Vote Count:                               3
  Reservation Key:                                 0x4979D88000000001
  Default Vote Count:                              3

Node Name:                                      c-220ra-2-epar03
  Node ID:                                         2
  Quorum Vote Count:                               1
  Reservation Key:                                 0x4979D88000000002


=== Quorum Devices ===                         

Quorum Device Name:                             qs
  Enabled:                                         yes
  Votes:                                           3
  Global Name:                                     qs
  Type:                                            quorum_server
  Hosts (enabled):                                 c-220ra-1-epar03, c-220ra-2-epar03
  Quorum Server Host:                              c-220ra-3-epar03
  Port:                                            9000

Quorum Device Name:                             d4
  Enabled:                                         yes
  Votes:                                           3
  Global Name:                                     /dev/did/rdsk/d4s2
  Type:                                            shared_disk
  Access Mode:                                     scsi3
  Hosts (enabled):                                 c-220ra-1-epar03, c-220ra-2-epar03

Custom installation with scinstall prompts for turning global fencing off



New fencing option versus local-only


Local only device group property previously used to turn fencing off Issues :

  • Takes a couple steps to turn fencing off

  • Not easy to turn fencing off for all devices

  • Requires the rawdisk device group to be configured

  • Doesn't work for quorum devices

Local only code is still in place

Local only still used for early-access VxVM disk groups


Turning fencing off for quorum devices



Quorum devices can be configured on devices that have fencing turned off. It Will use software quorum

Fencing cannot be changed for a configured quorum device. To change the fencing setting, the device must first be unconfigured as a quorum device. It is recommend using quorum server if fencing will be turned off and on regularly

c-220ra-1-epar03 #  cluster set -p global_fencing=nofencing
Warning: Device instance d4 is a quorum device - fencing protocol remains SCSI-3 for the device.
Updating shared devices on node 1
Updating shared devices on node 2



Changing the global fencing setting will change the setting for all devices, except quorum devices



CCR entries

( as a side note, CCR location have changed in  suncluster 3.2 update 2, now in /etc/cluster/ccr/global/ )



Per disk entry in did_instances


4       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE3967000001929HG1G|USE_GLOBAL|N/A 5345414741544520535433393130324c4353554e392e30474c4a453339363730303030303139323948473147|2:/dev/rdsk/c1t0d0|1:/dev/rdsk/c1t0d0
4       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE3967000001929HG1G|USE_SCSI3|N/A|5345414741544520535433393130324c4353554e392e30474c4a453339363730303030303139323948473147|1:/dev/rdsk/c1t0d0|2:/dev/rdsk/c1t0d0


Global setting is saved in global_fencing file

Only created if the global setting is changed from the default setting of pathcount

Uses numerical code for fencing setting, from libdid.h

#define GLOBAL_FENCING_UNKNOWN          1                               
#define GLOBAL_FENCING_PATHCOUNT        2                                       
#define GLOBAL_FENCING_PREFER3          3                                       
#define GLOBAL_NO_FENCING               4

troubleshooting


Check fencing setting :


c-220ra-1-epar03 # scdidadm -G

Fencing is currently disabled for disks using the global fencing setting.
c-220ra-1-epar03 # cldevice show

=== DID Device Instances ===                   

DID Device Name:                                /dev/did/rdsk/d1
  Full Device Path:                                c-220ra-1-epar03:/dev/rdsk/c0t0d0
  Replication:                                     none
  default_fencing:                                 global

DID Device Name:                                /dev/did/rdsk/d2
  Full Device Path:                                c-220ra-1-epar03:/dev/rdsk/c0t1d0
  Replication:                                     none
  default_fencing:                                 global

DID Device Name:                                /dev/did/rdsk/d3
  Full Device Path:                                c-220ra-1-epar03:/dev/rdsk/c0t6d0
  Replication:                                     none
  default_fencing:                                 global

DID Device Name:                                /dev/did/rdsk/d4
  Full Device Path:                                c-220ra-1-epar03:/dev/rdsk/c1t0d0
  Full Device Path:                                c-220ra-2-epar03:/dev/rdsk/c1t0d0
  Replication:                                     none
  default_fencing:                                 scsi3

DID Device Name:                                /dev/did/rdsk/d5
  Full Device Path:                                c-220ra-1-epar03:/dev/rdsk/c1t1d0
  Full Device Path:                                c-220ra-2-epar03:/dev/rdsk/c1t1d0
  Replication:                                     none
  default_fencing:                                 nofencing

...
DID Device Name:                                /dev/did/rdsk/d30
  Full Device Path:                                c-220ra-2-epar03:/dev/rdsk/c0t6d0
  Replication:                                     none
  default_fencing:                                 global

c-220ra-1-epar03 # clq show

=== Cluster Nodes ===                          

Node Name:                                      c-220ra-1-epar03
  Node ID:                                         1
  Quorum Vote Count:                               3
  Reservation Key:                                 0x4979D88000000001
  Default Vote Count:                              3

Node Name:                                      c-220ra-2-epar03
  Node ID:                                         2
  Quorum Vote Count:                               1
  Reservation Key:                                 0x4979D88000000002


=== Quorum Devices ===                         

Quorum Device Name:                             qs
  Enabled:                                         yes
  Votes:                                           3
  Global Name:                                     qs
  Type:                                            quorum_server
  Hosts (enabled):                                 c-220ra-1-epar03, c-220ra-2-epar03
  Quorum Server Host:                              c-220ra-3-epar03
  Port:                                            9000

Quorum Device Name:                             d4
  Enabled:                                         yes
  Votes:                                           3
  Global Name:                                     /dev/did/rdsk/d4s2
  Type:                                            shared_disk
  Access Mode:                                     scsi3
  Hosts (enabled):                                 c-220ra-1-epar03, c-220ra-2-epar03

Quorum Device Name:                             d6
  Enabled:                                         yes
  Votes:                                           3
  Global Name:                                     /dev/did/rdsk/d6s2
  Type:                                            shared_disk
  Access Mode:                                     sq_disk
  Hosts (enabled):                                 c-220ra-1-epar03, c-220ra-2-epar03


c-220ra-1-epar03 # /usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d4s2
Reservation keys(2):
0x4979d88000000001
0x4979d88000000002
c-220ra-1-epar03 # /usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d5s2
Reservation keys(0):

Make sure you don't have a quorum device configured if you are turning fencing off

Comments:

Thanks a lot JC for explanations,
I'm a storage guy migrating 2 clusters (SunCluster 3.2) to new storage.
I've noticed once cluster has his shared disks set to default_fencing: pathcount while the second cluster is set to "global" with the global_fencing: set to prefer3 for both. Both clusters are using the same storage array and will use after migration a new array model (but also both the same array). I could not find in the awailble documentation what are the implications of using pathcount for default fencing, what does actually and the local admin team doeas not know either why the 2 clusters had different settings.

Posted by guest on October 06, 2011 at 12:45 PM CEST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Jean-Christophe Lamoure

Search

Archives
« May 2015
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
      
Today