星期六 三月 03, 2012

Misleading example of 'Configuring Linux Initiators' in 'Sun ZFS Storage 7000 System Administration Guide'

For the past 2 weeks, I'd been busy in a Sun ZFS Storage 7420 cluster system installation project.

We'd spent much time to design and deploy. At last, seemed everything ok except multipath setting on Redhat 5.5 servers(with kernel-2.6.18-194.el5 and device-mapper-multipath-0.4.7-34.el5).

Every redhat server is connected to 2 SS7420 heads through 2 switches, and every redhat server should see every LUN through 4 path's, 2 active plus 2 standby.

Yet what we saw in multipath was not what we expected.

take one LUN as example:

======================================================================= 
pool1v0 (3600144f0ad85565d00004f4470760008) dm-21 SUN,ZFS Storage 7420
[size=4.0T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 8:0:1:14 sdau 66:224  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 6:0:3:14 sdeb 128:48  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 6:0:0:14 sdl  8:176   [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 8:0:0:14 sdq  65:0    [active][ready] 
========================================================================

Every path had same prio, and this was not correct.

At first, our configuration followed the one 'For OEL 5.5' because we thought Redhat 5.5 is (almost) same as OEL 5.5

========================================================================== 
devices {  
   device {  
     vendor                    "SUN"  
     product                   "Sun Storage <i>7420</i>"  
     getuid_callout            "/sbin/scsi_id -g -u -s /block/%n"  
     prio                      alua  
     hardware_handler          "1 alua"  
     path_grouping_policy      group_by_prio  
     failback                  immediate  
     no_path_retry             600  
     rr_min_io                 100  
     path_checker              tur  
     rr_weight                 uniform features "0"  
   }  

=====================================================================

Later we found that OEL 5.5 in E26397 is with below packages

=================================================
Unbreakable Enterprise Kernel 2.6.32-100.0.19.el5 
device-mapper-multipath: 0.4.9-23.0.9.el5 
kpartx: 0.4.9-23.0.9.el5 
=================================================

And OEL 5.4 in E26397 is with below packages

=================================================
kernel: 2.6.18-164.el5 
device-mapper-multipath: 0.4.7-30.el5 
kpartx: 0.4.7-30.el5 
=================================================

We believe it's the combination of kernel+device-mapper-multipath+kpartx that decides the configuration instead of linux release version.

Since we have same kernel+device-mapper-multipath+kpartx versions as that of OEL 5.4, we decided to use configuration for OEL 5.4

=============================================================
device 

    vendor                     "SUN" 
    product                    "Sun Storage <i>7420</i>" 
    getuid_callout             "/sbin/scsi_id -g -u -s /block/%n" 
    prio_callout               "/sbin/mpath_prio_alua /dev/%n" 
    hardware_handler           "0" 
    path_grouping_policy       group_by_prio 
    failback                   immediate 
    no_path_retry              queue 
    rr_min_io                  100 
    path_checker               tur 
    rr_weight                  uniform 
}
===============================================================

Still multipath showed same prio for all path's.

What's the problem? We'd tried configurations for both OEL 5.4 and 5.5.

Since E26397 is the only authorized document for interconnecting Linux and SS7000 series, we had to stick to it.

Re-read the document and we found an emphatic note

==========================================================================
NOTE: replace <i>NNNN</i> with the appropriate model number -- e.g., 7420.
==========================================================================

When you first see this line, don't you think it asks you to replace 'NNNN' with '7420' instead of '<i>NNNN</i>' with '7420'?


Don't lie to me, I bet you have the same thought.


So we should replace 'product' line with below

=================================================
product             "Sun Storage 7420" 
================================================

Are we ok now? No!

'multipath -v6 -ll' shows it does NOT use 'prio_callout' in our configuration

==============================================
...
sdbn: mask = 0x8
sdbn: getprio = NULL (internal default)
sdbn: prio = 1
sdch: mask = 0x8
sdch: getprio = NULL (internal default)
sdch: prio = 1
sdcm: mask = 0x8
sdcm: getprio = NULL (internal default)
sdcm: prio = 1
sddk: mask = 0x8
sddk: getprio = NULL (internal default)
sddk: prio = 1
...
============================================

'(internal default)' instead of '(config file default)'.


Our config file does not take effect?

What on earth is the problem?

Well, we should have found the problem earlier should we observed the 'multipath -v6 -ll' output carefully.

==========================================================
...
sdbn: not found in pathvec
sdbn: mask = 0x5
sdbn: bus = 1
sdbn: dev_t = 68:16
sdbn: size = 8589934592
sdbn: vendor = SUN
sdbn: product = ZFS Storage 7420
sdbn: rev = 1.0
sdbn: h:b:t:l = 8:0:2:13
sdbn: tgt_node_name = 0x20000024ff2d5ce8
sdbn: path checker = readsector0 (config file default)
sdbn: checker timeout = 60000 ms (sysfs setting)
sdbn: state = 2
...
==========================================================

Can you find the clues without my highlight?

The 'product' string should be 'ZFS Storage 7420' instead of 'Sun Storage 7420'? 

What does this matter? It matters all.

The 'vendor' and 'product' combination is NOT just a description, it's the key for multipath to match a section in the config file to the devices it detects.

If we specify wrong 'vendor' or 'product', we will not get what we expect since there's no match.

So here comes our correct confg 

=============================================================
device 

    vendor                     "SUN" 
    product                    "ZFS Storage 7420
    getuid_callout             "/sbin/scsi_id -g -u -s /block/%n" 
    prio_callout               "/sbin/mpath_prio_alua /dev/%n" 
    hardware_handler           "0" 
    path_grouping_policy       group_by_prio 
    failback                   immediate 
    no_path_retry              queue 
    rr_min_io                  100 
    path_checker               tur 
    rr_weight                  uniform 
}
===============================================================

And now multipath shows correct prio of 4 path's

=====================================
pool1v1 (3600144f0ad85565d00222f447076001a) dm-24 SUN,ZFS Storage 7420  [size=4.0T][features=0][hwhandler=0][rw] 
\_ round-robin 0 [prio=100][enabled] 
 \_ 8:0:1:34 sdbe 67:128  [active][ready] 
 \_ 6:0:3:34 sdel 128:208 [active][ready] 
\_ round-robin 0 [prio=2][enabled] 
 \_ 6:0:0:34 sdaf 65:240  [active][ready] 
 \_ 8:0:0:34 sdak 66:64   [active][ready] 
===================================================

Why doesn't E26397 say we need change this? 

I guess this manual is for 7x10 because 7x10 has the name of 'SUN Storage ...', as for 7x20, they change to 'ZFS Storage ...'.



PS:




-EOF-

About

samwan

Search

Categories
Archives
« 三月 2012
星期日星期一星期二星期三星期四星期五星期六
    
1
2
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
       
今天