Device ID

Purpose of the did layer : provide global access to devices irrespective of their physical location.
Global Naming (DID Devices)

/dev/did/dsk and /dev/did/rdsk

  • DID device names are not used in VxVM

  • DID device names are used in Sun/Solaris Volume Manager

Global Devices
  • provide global access to devices irrespective of their physical location.

/global/.devices/node@nodeID

nodeID is an integer representing the node in the cluster (/etc/cluster/nodeid).

Device names are created in /dev/did/dsk and /dev/did/rdsk in groups of 100 disks at a time. For tapes, device names are created in /dev/did/rmt in groups of 10 tapes at a time. If more devices are added to the cluster than are handled by the current names, another group will be created.

FILES :


/dev/did/dsk/dXsY

Bloc disk or CDROM device, where X is the device number and Y is the slice number

/dev/did/rdsk/dXsY

Raw disk or CDROM device, where X is the device number and Y is the slice number

/dev/did/rmt/N

Tape device, where N is the device number

/dev/did/admin

Administrative device

/kernel/drv/did

Driver module

/kernel/drv/did.conf

Driver configuration file



[c-220ra-2-epar03][root(18)] scdidadm -L
1        c-220ra-1-epar03:/dev/rdsk/c0t0d0 /dev/did/rdsk/d1     
2        c-220ra-1-epar03:/dev/rdsk/c0t1d0 /dev/did/rdsk/d2     
3        c-220ra-1-epar03:/dev/rdsk/c0t6d0 /dev/did/rdsk/d3     
4        c-220ra-1-epar03:/dev/rdsk/c1t0d0 /dev/did/rdsk/d4     
4        c-220ra-2-epar03:/dev/rdsk/c1t0d0 /dev/did/rdsk/d4     
5        c-220ra-1-epar03:/dev/rdsk/c1t1d0 /dev/did/rdsk/d5     
5        c-220ra-2-epar03:/dev/rdsk/c1t1d0 /dev/did/rdsk/d5     
6        c-220ra-1-epar03:/dev/rdsk/c1t2d0 /dev/did/rdsk/d6     
6        c-220ra-2-epar03:/dev/rdsk/c1t2d0 /dev/did/rdsk/d6     
7        c-220ra-1-epar03:/dev/rdsk/c1t3d0 /dev/did/rdsk/d7     
7        c-220ra-2-epar03:/dev/rdsk/c1t3d0 /dev/did/rdsk/d7     
8        c-220ra-1-epar03:/dev/rdsk/c1t4d0 /dev/did/rdsk/d8     
8        c-220ra-2-epar03:/dev/rdsk/c1t4d0 /dev/did/rdsk/d8     
9        c-220ra-1-epar03:/dev/rdsk/c1t5d0 /dev/did/rdsk/d9     
9        c-220ra-2-epar03:/dev/rdsk/c1t5d0 /dev/did/rdsk/d9     
10       c-220ra-1-epar03:/dev/rdsk/c1t8d0 /dev/did/rdsk/d10    
10       c-220ra-2-epar03:/dev/rdsk/c1t8d0 /dev/did/rdsk/d10    
11       c-220ra-1-epar03:/dev/rdsk/c1t9d0 /dev/did/rdsk/d11    
11       c-220ra-2-epar03:/dev/rdsk/c1t9d0 /dev/did/rdsk/d11    
...

[c-220ra-2-epar03][root(22)] cldevice list -v
DID Device          Full Device Path
----------          ----------------
d1                  c-220ra-1-epar03:/dev/rdsk/c0t0d0
d2                  c-220ra-1-epar03:/dev/rdsk/c0t1d0
d3                  c-220ra-1-epar03:/dev/rdsk/c0t6d0
d4                  c-220ra-1-epar03:/dev/rdsk/c1t0d0
d4                  c-220ra-2-epar03:/dev/rdsk/c1t0d0
d5                  c-220ra-1-epar03:/dev/rdsk/c1t1d0
d5                  c-220ra-2-epar03:/dev/rdsk/c1t1d0
d6                  c-220ra-1-epar03:/dev/rdsk/c1t2d0
d6                  c-220ra-2-epar03:/dev/rdsk/c1t2d0
d7                  c-220ra-1-epar03:/dev/rdsk/c1t3d0
d7                  c-220ra-2-epar03:/dev/rdsk/c1t3d0
d8                  c-220ra-1-epar03:/dev/rdsk/c1t4d0
d8                  c-220ra-2-epar03:/dev/rdsk/c1t4d0
d9                  c-220ra-1-epar03:/dev/rdsk/c1t5d0
d9                  c-220ra-2-epar03:/dev/rdsk/c1t5d0
d10                 c-220ra-1-epar03:/dev/rdsk/c1t8d0
d10                 c-220ra-2-epar03:/dev/rdsk/c1t8d0
d11                 c-220ra-1-epar03:/dev/rdsk/c1t9d0
d11                 c-220ra-2-epar03:/dev/rdsk/c1t9d0
...

/etc/cluster/ccr/did_instances corresponding to above devices :


[c-220ra-2-epar03][root(28)] cat did_instances

ccr_gennum      3
ccr_checksum    E2BA23BADE85E6CF942CC4FA5BD7E51A

30      disk|||USE_GLOBAL|N/A||2:/dev/rdsk/c0t6d0
1       disk|DEVID_SCSI_SERIAL|SEAGATE ST336607LSUN36G 3JAX059V00007337K2HT|USE_GLOBAL|N/A|534541474154452053543333363630374c53554e33364720334a41583035395630303030373333374b324854|1:/dev/rdsk/c0t0d0
2       disk|DEVID_SCSI3_VPD_T10|HITACHI DK32EJ-36NC     433V4927|USE_GLOBAL|N/A|4849544143484920444b3332454a2d33364e4320202020203433335634393237|1:/dev/rdsk/c0t1d0
3       disk|||USE_GLOBAL|N/A||1:/dev/rdsk/c0t6d0
4       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE3967000001929HG1G|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a453339363730303030303139323948473147|2:/dev/rdsk/c1t0d0|1:/dev/rdsk/c1t0d0
5       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE38325000019290LVE|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a4533383332353030303031393239304c5645|2:/dev/rdsk/c1t1d0|1:/dev/rdsk/c1t1d0
6       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE22272000029290C8E|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a453232323732303030303239323930433845|2:/dev/rdsk/c1t2d0|1:/dev/rdsk/c1t2d0
7       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE2633400001929HJ81|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a4532363333343030303031393239484a3831|2:/dev/rdsk/c1t3d0|1:/dev/rdsk/c1t3d0
8       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE1946500001929HEJW|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a453139343635303030303139323948454a57|2:/dev/rdsk/c1t4d0|1:/dev/rdsk/c1t4d0
9       disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJG2894300001931J9QB|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a47323839343330303030313933314a395142|2:/dev/rdsk/c1t5d0|1:/dev/rdsk/c1t5d0
10      disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE22518000019290PZG|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47|2:/dev/rdsk/c1t8d0|1:/dev/rdsk/c1t8d0
11      disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE35262000019290787|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a453335323632303030303139323930373837|2:/dev/rdsk/c1t9d0|1:/dev/rdsk/c1t9d0

...


let's have a closer look at that file structure :

a line is composed by a number of fields :


10      disk|DEVID_SCSI_SERIAL|SEAGATE ST39102LCSUN9.0GLJE22518000019290PZG|USE_GLOBAL|N/A|5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47|2:/dev/rdsk/c1t8d0|1:/dev/rdsk/c1t8d0

the field delimiter is the “|” character.

So we have for the line above :


10

disk

DEVID_SCSI_SERIAL

SEAGATE ST39102LCSUN9.0GLJE22518000019290PZG

USE_GLOBAL

N/A

5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47

2:/dev/rdsk/c1t8d0

1:/dev/rdsk/c1t8d0


Field

Example

Explanation

#

10

The did device number

type

disk

The type of device (disk, tape) defined in the did_types table

Device Class

DEVID_SCSI_SERIAL

type of device ID that this device provides. If none is given by the device, this field will be empty. Possible values : DEVID_SCSI3_WWN or DEVID_SCSI_SERIAL DEVID_SCSI3_VPD_\* ...

Vendor info

SEAGATE ST39102LCSUN9.0GLJE22518000019290PZG

An ascii representation of the device ID. For some devices, such as Fiber Channel disks, this field will be empty.


USE_GLOBAL



N/A


Unique Id

5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47

A representation of the device ID in hex. This is the definitive representation, and the ID used to compare devices amongst nodes. For some classes of devices (e.g. tapes or CDROMs), there will be no device ID

Subpath

2:/dev/rdsk/c1t8d0

1:/dev/rdsk/c1t8d0

<hostname>:<device path> There is one subpath entry per physic al path to the device.



The seventh field is the raw value of the diskid (5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47)

this is the hexadecimal raw value. You can translate it to the ascii value knowing that 0x41=A, 0x42=B etc)



"DEVID_SCSI3_WWN does not relate to whether the cluster will use scsi2 or scsi3 protocol for the Quorum keys.It relates to whether the disk device supports the VPD (Vital Product Data) page 0x83. If you want further information on this, then check out the SCSI protocol documents.

VPD page 0x83 was only introduced a few years ago when Fiber Channel devices were first brought out.

In cluster it checks if the disk device has a VPD page 0x83, if so then sets the "Device ID Type" to be DEVID_SCSI3_WWN. If the device does not have a VPD page 0x83, cluster then checks if the disk has been qualified by the indication of the string "SUN" in the contents of an INQUIRY command to the device. If it does contain the string "SUN", it's marked as a DEVID_SCSI_SERIAL device.

The number of paths determines whether we are going to use scsi-2 or scsi3.


now let's have a closer look to c1t8d0 :


format output :


c1t8d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
          /pci@1f,4000/scsi@2/sd@8,0

sd10      Soft Errors: 0 Hard Errors: 1 Transport Errors: 0 
Vendor: SEAGATE  Product: ST39102LCSUN9.0G Revision: 0828 Serial No: 9904E37404 
Size: 9.06GB <9055065600 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 


scdidadm -k :


#
# SEAGATE ST39102LCSUN9.0GLJE22518000019290PZG
#
instance        10
        type DEVID_SCSI_SERIAL
        diskid 5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47
        c-220ra-2-epar03:/dev/dsk/c1t8d0
#


let's look at a more complex config (from an explorer) :



/cluster/did/scdidadm-L.out


10       cpsapp:/dev/rdsk/c4t5006048C49AF7887d26 /dev/did/rdsk/d10
10       cpsdb:/dev/rdsk/c6t5006048C49AF7887d26 /dev/did/rdsk/d10
10       cpsdb:/dev/rdsk/c3t5006048C49AF7888d26 /dev/did/rdsk/d10
10       cpsapp:/dev/rdsk/c7t5006048C49AF7888d26 /dev/did/rdsk/d10
11       cpsapp:/dev/rdsk/c4t5006048C49AF7887d25 /dev/did/rdsk/d11
11       cpsdb:/dev/rdsk/c6t5006048C49AF7887d25 /dev/did/rdsk/d11
11       cpsdb:/dev/rdsk/c3t5006048C49AF7888d25 /dev/did/rdsk/d11
11       cpsapp:/dev/rdsk/c7t5006048C49AF7888d25 /dev/did/rdsk/d11


/cluster/etc/cluster/ccr/did_instances


10      disk|DEVID_SCSI3_VPD_NAA|`|USE_GLOBAL|N/A|60060480000287751650524144333342|2:/dev/rdsk/c7t5006048C49AF7888d26|1:/dev/rdsk/c3t5006048C49AF7
888d26|1:/dev/rdsk/c6t5006048C49AF7887d26|2:/dev/rdsk/c4t5006048C49AF7887d26
11      disk|DEVID_SCSI3_VPD_NAA|`|USE_GLOBAL|N/A|60060480000287751650524144333333|2:/dev/rdsk/c7t5006048C49AF7888d25|1:/dev/rdsk/c3t5006048C49AF7
888d25|1:/dev/rdsk/c6t5006048C49AF7887d25|2:/dev/rdsk/c4t5006048C49AF7887d25

so lets' look at d10 :




Field

Example

Explanation

#

10

The did device number

type

disk

The type of device (disk, tape) defined in the did_types table

Device Class

DEVID_SCSI3_VPD_NAA

type of device ID that this device provides. If none is given by the device, this field will be empty. Possible values : DEVID_SCSI3_WWN or DEVID_SCSI_SERIAL DEVID_SCSI3_VPD_\* ...

Vendor info

`

An ascii representation of the device ID. For some devices, such as Fiber Channel disks, this field will be empty.


USE_GLOBAL



N/A


Unique Id

60060480000287751650524144333342

A representation of the device ID in hex. This is the definitive representation, and the ID used to compare devices amongst nodes. For some classes of devices (e.g. tapes or CDROMs), there will be no device ID

Subpath

2:/dev/rdsk/c7t5006048C49AF7888d26

1:/dev/rdsk/c3t5006048C49AF7888d26

1:/dev/rdsk/c6t5006048C49AF7887d26

2:/dev/rdsk/c4t5006048C49AF7887d26

<hostname>:<device path> There is one subpath entry per physic al path to the device.



disks/diskinfo :


c7t5006048C49AF7888d26 EMC SYMMETRIX 5670 5033B000Q primary

c4t5006048C49AF7887d26 EMC SYMMETRIX 5670 5033B000Q primary


disk/format.out :


28. c4t5006048C49AF7887d26 <EMC-SYMMETRIX-5670 cyl 37998 alt 2 hd 60 sec 64>

/pci@5,700000/SUNW,qlc@0,1/fp@0,0/ssd@w5006048c49af7887,1a


62. c7t5006048C49AF7888d26 <EMC-SYMMETRIX-5670 cyl 37998 alt 2 hd 60 sec 64>

/pci@5,700000/SUNW,qlc@0,1/fp@0,0/ssd@w5006048c49af7888,1a






Decoding a unique id :

[jfk][jclamour(128)] did_decode 5345414741544520535433393130324c4353554e392e30474c4a453232353138303030303139323930505a47
SEAGATE ST39102LCSUN9.0GLJE22518000019290PZG
[jfk][jclamour(129)] did_decode 60060480000287751650524144333342
`uPRAD33B

did_decode.c :

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>



int main (int argc, char \*argv[])
        {
        int i=0;
        char chaine [500];
        char hex[3];
        if (argc!=2)
                {
                fprintf(stderr,"usage : %s <string to decode.\\n",argv[0]);
                return(-1);
                }

        strcpy(chaine,argv[1]);
        for (i=0;i<strlen(chaine); i+=2)
                {
                sprintf(hex,"%c%c",chaine[i],chaine[i+1]);
                hex[2]='\\0';
                printf("%c",strtoul(hex,NULL,16)); 
                }
        printf("\\n");
        return(0);
        }



Data path :



Application (userland)

SVM

DID

Disk driver (sd,ssd,..)

Storage HW



So any trouble on the did layer can damage/destroy the data path.


Note : with Veritas vxvm, did are not used in the datapath, but they are still used for Quorum device access.



Operations “at risk” that may change the diskid :

  • replacing a disk in a JBOD

  • destroying/recreating a LUN


Operations that are usually neutral regarding diskid :

  • HBA replacement

  • Disk replacement within a LUN ( without LUN re creation)



Commands :


SC 3.0 3.1 : scdidadm

SC 3.2 : cldevice, scdidadm

Some documents to correctly interact with the DID layer :

Solution  206944 :   Sun[TM] Cluster 3.x: How to change SCSI JBOD disk with Solstice DiskSuite[TM] SDS // Solaris Volume Manager[TM] SVM  

Solution  202778 :   Sun[TM] Cluster 3.x: How to change SCSI JBOD disk under Veritas Volume Manager 3.x  

Solution  207187 :   Sun[TM] Cluster 3.x: How to recover did instances when devices are changed  

Solution  210631 :   Sun[TM] Cluster 3.0 and 3.1: How to fix wrong DID number after a disk replacement  








Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Jean-Christophe Lamoure

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today