Transparent Failover with Solaris MPxIO and Oracle ASM

Recently I had a nice experience configuring failover with Solaris MPxIO (Multipathing), Oracle ASM (Automatic Storage Management) and Oracle's Sun Storage 6180 array. In this configuration failover was completely transparent to the Oracle database.

MPxIO is part of the Solaris 10 distribution. Multipathing provides redundant paths and eliminates a single point of failure by failing over to an alternate path automatically when one of the paths fails. Oracle ASM was used for volume manager to manage Oracle database files and it works very well with Multipathing. The Sun Storage 6180 array configuration had two controllers and each controller was connected to a different adapter on the host for redundant path. Multipathing generates a pseudo device and by providing Oracle ASM a pseudo device name at ASM diskgroup creation it makes failover transparent to the Oracle database.

HW/SW Configuration

Sun SPARC Enterprise M9000, 2x 8Gb FC HBA
1xSun Storage 6180 array, 16x 300G 15K RPM disks, two controllers, Firmware 07.60.18.13
Solaris 10, Oracle 11g with Oracle ASM
MPxIO enabled

When one of the paths fails (e.g. Cable, adapter, controller) system messages in /var/adm/messages file show the following sequence of events.

    Link down      <- the path is down
    offlining lun= ...       <- associated devices are offline
    multipath status: degraded      <- multipath status is degraded
    Initiating failover for device       <- Initiate failover
    Failover operation completed successfully      <- failover was successful
However during this period Oracle database transactions continue without any interruption and the Oracle database doesn't even know this is happening. The failover is transparent to the Oracle database and there was no error or message in Oracle log files during this period.

The following shows the steps and examples to set up MPxIO and Oracle ASM. At the bottom of the page it shows system messages from failover.

Sun Storage 6180 array

First you need to configure two paths physically. In this case, one controller was connected to one of the HBAs on the host and the other controller was connected to the other HBA.


Use Sun StorageTEK Common Array Manager GUI tool or command tool 'sscs' (StorEdge Systems Element Manager Command Line Interface) to manage 6180.
6180 'os-type' needs to be set properly to support MPxIO.
To view the current 'os-type' setting run 'sscs list array array_name'.
'Default Host Type' should be 'SOLARIS_MPXIO - Solaris (with Traffic Manager)'.
Run 'sscs list -a array_name os-type' to view Operating Systems supported and 'sscs modify -o os-type_name array array_name' to modify 'os-type'.

Host

Solaris 'stmsboot -e' command enables MPxIO. It will prompt you to reboot the host.


After the reboot 'format' shows a long pseudo name such as c29t60080E500018142E000003574C5 EBF38d0
But 'stmsboot -L' provides both before and after MPxIO. This mapping is very useful for any debugging you might need.

#stmsboot -L

 non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c6t0d0        /dev/rdsk/c29t60080E5000184350000003614C5EBE23d0
/dev/rdsk/c6t0d2        /dev/rdsk/c29t60080E5000184350000003654C5EBE44d0
/dev/rdsk/c6t0d1        /dev/rdsk/c29t60080E50001845C6000003524C5EBE2Ed0
/dev/rdsk/c6t0d3        /dev/rdsk/c29t60080E50001845C6000003564C5EBE4Cd0
/dev/rdsk/c9t4d0        /dev/rdsk/c29t60080E5000184350000003614C5EBE23d0
/dev/rdsk/c9t4d2        /dev/rdsk/c29t60080E5000184350000003654C5EBE44d0
/dev/rdsk/c9t4d1        /dev/rdsk/c29t60080E50001845C6000003524C5EBE2Ed0
/dev/rdsk/c9t4d3        /dev/rdsk/c29t60080E50001845C6000003564C5EBE4Cd0
Notice c6t0 and c9t4 point to the same STMS device name (e.g. c6t0d0 and c9t4d0 point to c29t60080E5000184350000003614C5EBE23d0).

Oracle ASM

Create a ASM diskgroup by providing MPxIO pseudo device names.

Initialize disks by doing 'dd if=/dev/zero of=/dev/rdsk/c29t60080E50001843500 00003614C5EBE23d0s0 bs=1204k count=10.


Set owner to Oracle 'chown oracle:dba /dev/rdsk/c29t60080E500018435000000 3614C5EBE23d0s0'
create diskgroup diskgroup_name external redundancy disk
   '/dev/rdsk/c29t60080E5000184350000003614C5EBE23d0s0' size 10g,
   '/dev/rdsk/c29t60080E5000184350000003654C5EBE44d0s0' size 10g,
   '/dev/rdsk/c29t60080E50001845C6000003524C5EBE2Ed0s0' size 10g,
   '/dev/rdsk/c29t60080E50001845C6000003564C5EBE4Cd0s0' size 10g;

System messages from failover

When one of the paths fails at the array side 'sscs list -a array_name fcport ' shows one of the ports is down.

And /var/adm/messages file reports:

Aug 10 11:17:33 host1 emlxs: [ID 349649 kern.info] [ 5.0314]emlxs0: NOTICE: 710: Link down.
Aug 10 11:19:03 host1 fctl: [ID 517869 kern.warning] WARNING: fp(2)::OFFLINE timeout
Aug 10 11:19:22 host1 scsi: [ID 243001 kern.info] /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fcp2):
Aug 10 11:19:22 host1    offlining lun=3 (trace=0), target=ef (trace=2800004)
Aug 10 11:19:22 host1 scsi: [ID 243001 kern.info] /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fcp2):
Aug 10 11:19:22 host1    offlining lun=2 (trace=0), target=ef (trace=2800004)
Aug 10 11:19:22 host1 scsi: [ID 243001 kern.info] /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fcp2):
Aug 10 11:19:22 host1    offlining lun=1 (trace=0), target=ef (trace=2800004)
Aug 10 11:19:22 host1 scsi: [ID 243001 kern.info] /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fcp2):
Aug 10 11:19:22 host1    offlining lun=0 (trace=0), target=ef (trace=2800004)
Aug 10 11:19:22 host1 genunix: [ID 408114 kern.info] /pci@6,600000/SUNW,emlxs@0/fp@0,0/ssd@w2014
0080e51845c6,1f (ssd0) offline
Aug 10 11:19:22 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e50001845c6000003564c5
ebe4c (ssd165) multipath status: degraded, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to targe
t address: w20140080e51845c6,3 is offline Load balancing: round-robin
Aug 10 11:19:22 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e5000184350000003654c5
ebe44 (ssd166) multipath status: degraded, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to targe
t address: w20140080e51845c6,2 is offline Load balancing: round-robin
Aug 10 11:19:22 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e50001845c6000003524c5
ebe2e (ssd167) multipath status: degraded, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to targe
t address: w20140080e51845c6,1 is offline Load balancing: round-robin
Aug 10 11:19:22 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e5000184350000003614c5
ebe23 (ssd168) multipath status: degraded, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to targe
t address: w20140080e51845c6,0 is offline Load balancing: round-robin
Aug 10 11:19:22 host1 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Aug 10 11:19:22 host1    Initiating failover for device ssd (GUID 60080e5000184350000003614c5ebe
23)
Aug 10 11:19:23 host1 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Aug 10 11:19:23 host1    Failover operation completed successfully for device ssd (GUID 60080e50
00184350000003614c5ebe23): failed over from primary to secondary
Aug 10 11:23:10 host1 emlxs: [ID 349649 kern.info] [ 5.0536]emlxs0: NOTICE: 720: Link up. (8Gb,
loop, initiator)
Aug 10 11:23:10 host1 genunix: [ID 936769 kern.info] ssd0 is /pci@6,600000/SUNW,emlxs@0/fp@0,0/s
sd@w20140080e51845c6,1f
Aug 10 11:23:11 host1 genunix: [ID 408114 kern.info] /pci@6,600000/SUNW,emlxs@0/fp@0,0/ssd@w2014
0080e51845c6,1f (ssd0) online
Aug 10 11:23:11 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e50001845c6000003564c5
ebe4c (ssd165) multipath status: optimal, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to target
 address: w20140080e51845c6,3 is standby Load balancing: round-robin
Aug 10 11:23:11 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e5000184350000003654c5
ebe44 (ssd166) multipath status: optimal, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to target
 address: w20140080e51845c6,2 is online Load balancing: round-robin
Aug 10 11:23:11 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e50001845c6000003524c5
ebe2e (ssd167) multipath status: optimal, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to target
 address: w20140080e51845c6,1 is standby Load balancing: round-robin
Aug 10 11:23:11 host1 genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60080e5000184350000003614c5
ebe23 (ssd168) multipath status: optimal, path /pci@6,600000/SUNW,emlxs@0/fp@0,0 (fp2) to target
 address: w20140080e51845c6,0 is standby Load balancing: round-robin
Aug 10 11:23:11 host1 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Aug 10 11:23:11 host1    Initiating failover for device ssd (GUID 60080e5000184350000003614c5ebe
23)
Aug 10 11:23:13 host1    Failover operation completed successfully for device ssd (GUID 60080e50
00184350000003614c5ebe23): failed over from secondary to primary
Aug 10 11:23:13 host1 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Aug 10 11:23:13 host1    Auto failback operation succeeded for device ssd (GUID 60080e5000184350
000003614c5ebe23)
That's it. Try this for yourself.
Comments:

Post a Comment:
Comments are closed for this entry.
About

BestPerf is the source of Oracle performance expertise. In this blog, Oracle's Strategic Applications Engineering group explores Oracle's performance results and shares best practices learned from working on Enterprise-wide Applications.

Index Pages
Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today