SVM default interlace and resync buffer values

open_solaris_blog

SVM default interlace and resync buffer values

Quick confession
I'm a newbie and a recent blogging convert :-) Writing has never been my biggest hobby though I really enjoy reading. However, OpenSolaris and seeing all the beneficial information/conversations generated from fellow Sun employees' blogs inspired me to write a short one.

Intro
I'm Tony Nguyen, another engineer in the Solaris Volume Manager team. Btw, the SVM team is a wonderful collection of engineers/people and I really enjoy being a part of the team. Anyway, there seems to be a common misperception about SVM performance(with the default interlace and resync buffer values) that we hear from time to time. While there's current work to modify these default values to achieve more optimal default performance, sysadmins can certainly get the improved performance with some simple modifications to how they create SVM metadevices. This blog will discuss how increasing default stripes and raid5 interlace values and mirror resync buffer size to improve the overall SVM performance.

Current Values
The current default interlace size for both stripe and RAID(raid5) metadevices is 16K or 32blks. This interlace value implies data is stored in 16K chunks and the consecutive 16K chunks spread across subcomponents(columns of a RAID or components of a stripe) in a metadevice. In processing I/O, SVM reads/writes interlace amount of data from each component simultaneously. The current resync buffer size is set at 64K or 128blks. The resync buffer size is the amount of data transferred between one submirror(or submirror component) to another submirror when SVM performs data synchronization for a mirror/raid1 metadevice(e.g attaching of a submirror).

Now, the 16K and 64K values for interlace and resync buffer, respectively, are due to the I/O capability of both older hardware and Solaris releases. It's quite obvious that small interlace and resync buffer values would produce a high number of I/O operations for large I/O sizes since the I/O is done in small chunks. Naturally, increasing the interlace and resync buffer values would improve the I/O and resync performance. But how large should these values be?

Suggested Values
We've conducted some performance testing with values ranging from 32K to 1024K for both the interlace and resync buffer sizes. The results showed the 512K interlace and resync buffer size seems to give the optimal additional performance, almost twice the throughput when I/O size is greater than 16K and half the submirror resync time(when metattaching submirrors or hotsparing components). The resync buffer size of 1024k gives only an additional 4-8% resync performance compared 512k buffer size. So how does one use these new default values?

    Specify the 512k interlace value when creating metadevices:
        RAID(raid5): metainit d5 -r cxtxdxsx cytydysy cztzdzsz -i 512k
        Stripe(raid0): metainit d0 3 1 cxtxdxsx 1 cytydysy cztzdzsz -i 512k

    Changing the resync buffer size
        - Edit /etc/system to specify: set md_mirror:md_resync_bufsz = 1024
        - Reboot the machine

Let me know if you found this blog to be of some use or would like to know about some other areas in SVM. By the way, other SVM engineers who've posted excellent blogs are:
Sanjay Nadkarni
Jerry Jelinek
Steve Peng
Susan Kamm-Worrell
Alok Aggarwal
And the team plans to post more SVM related blogs in the near future, stay tuned.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris
Comments:

Regarding RAID-5 write performance:


 
What about full line writes? The larger the interlace (segment) size the lesser the probability to have full line writes (esp. ontop of filesystems).
 
I once tried large interlace values and write performance really sucked. Next time I might even try smaller values.

Posted by Daniel Rock on September 05, 2005 at 07:15 AM PDT #

This is 3.5 years only. Any updated information?
How about Raid-0 with a large interlace size compared to Raid-5 in a 100% sequential write environment.
Then toss in NFS transfer on top of that.
We have a large Oracle database and some files will be spun off as Oracle grows during updating in a test lab environment. Since not enough local, san, nas disk (temp problem), we were going to write over NFS to a large, striped meta disk, and really this article only addresses raid-5, not raid-0 performance. How about 100% writes over NFS to raid-0. Can you compare to raid-5? Would a 1024kb interlace size still be best. How much faster than Raid-5 i/o with the parity read/write overhead eliminated?
Thanks.
Steve

Posted by Steve Kyes on December 16, 2008 at 09:41 AM PST #

thankxxxxxxxxx for nice support

Posted by sree on September 18, 2009 at 12:32 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

tonyn

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today