Tuesday May 27, 2008

Making SAMBA Go Faster.....

I do a lot of work with the CIFS server that we now provide as part of OpenSolaris, but I also still do work with SAMBA as well.

I have been experimenting with a workload where I am accessing 100 files from a Windows Server 2003 CIFS client (Sun Fire V40z). The server is a Sun Fire X4500 running SAMBA. I am doing sequential I/O with a  workload generating tool, and at any time 75 files are open for read and 25 open for write. There are 1million files in the file system comprising 4 TB of data.

I was not seeing great scalability or performance, then some research by a colleague (supported by some folk at samba.org) led me to try enabling Async IO (AIO) on the SAMBA server. This is a standard feature of SAMBA, and has been available in Sun's SAMBA server build since Solaris 10 8/07. From what I have been told , AIO specifically helps the case of client workload scaling w/ threads Vs scaling with many connections; the workload generating tool I am using (vdbench) scales with threads.

To enable AIO add these lines to the global part of your smb.conf :

aio read size = 1
aio write size =1

Then restart SAMBA.

This parameter is defined in bytes. This setting means that any I/O over 1 byte will be handled asynchronously by smbd. There may be reasons for this to be a bigger value in some cases, I don't know.

Without AIO, my previous best result was 42 MB/sec reads and 14 MB/sec writes; with AIO the client could read at 64.4 MB/sec reads and write a 22.1 MB/sec. CPU utilization on the X4500 running SAMBA went up from 15% to 50%.

Your mileage will vary depending on your workload and application, but that is quite a nice boost for just adding two lines to a configuration file :-)

Note: This work was done running the Solaris Express Community Edition snv_89 X86 (aka Nevada) on the Sun Fire X4500. The underlying file system was a ZFS file system provisioned from a RAID-Z storage pool configured as described here. I was reading/writing 8KB blocks.

Monday Jan 14, 2008

Using the Sun Fire X4500 as Network Attached Archival Storage

A paper about a project I completed recently has been published on BigAdmin. The paper is "Configuring the Sun Fire X4500 Server as Network Attached Archival Storage for Symantec Enterprise Vault". The aim of the project was to see if the Sun Fire X4500 could be used as NAS in a Symantec Enterprise Vault environment. The findings were that this worked very well, and performance was excellent!

Software used on the Sun Fire X4500 was SAMBA, Sun StorageTek Storage Archive Manager, and Solaris ZFS.The project involved functional and performance testing, both of which are part of Symantec's comprehensive Self Certification program for Partners.

I went the extra mile, and wrote a configuration tool for the solution called x4500samconfig. The use of the tool is discussed in the paper, and it is available for download here. If you take an X4500 with Solaris 10 8/07 and the unconfigured Storage Archive Manager 4.6 packages installed, this script will have you up and running in minutes. The script configures ZFS, Storage Archive Manager and SAMBA. Some of what the script does "under the hood" is discussed here.

I am going to take the opportunity here to thank Sun's BigAdmin team: the process of publishing an article like this is quite lengthy. Initially, a paper is written by an Engineer (me in this case) for Sun Internal consumption only...this is pretty straightforward, requiring peer review only. Once the internal version is complete, the paper can be submitted for external publication on BigAdmin, and at this point the document must be legally reviewed and professionally edited which can take some time...and it is the BigAdmin team who do this...so, Thanks BigAdmin Team!

Monday Nov 05, 2007

SMB/CIFS in Solaris and What It Means to SAMBA

This blog entry is worth reading if you are interested in the work Sun is doing to put SMB/CIFS into the Solaris Kernel. there is also a presentation here.

Tuesday Oct 09, 2007

Another SAMBA Test on Sun Fire X4500 (With fewer disks)

I received a few emails from the zfs-discuss@opensolaris.org alias about the results I posted yesterday...I started to feel curious about what would happen if I had fewer disks in the ZFS storage pool. So today I ran just the RAID-Z test again, but with half the number of disks in the ZFS storage pool shared by SAMBA.

The equipment used was as before except that the Sun Fire X4600 was replaced with a  Sun Fire X4200. 

The pool was created as below:

zpool create -f  sambapool \\
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \\
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
spare c0t1d0 c1t2d0 c4t3d0

When I looked at the results, I was at first I very puzzled: the performance (i.e. Total IOPS and Data Rate for the Sun Fire X4500) was identical to the previous test with twice as many disks.

On closer inspection I saw that each individual disk in the storage pool did twice as much work to achieve this vs the earlier test, so the mystery was solved.

Here are the graphs for c0t0d0, for comparison with the previous results:

RAID-Z c0t0d0 Data Rate


RAID-Z c0t0d0 IOPS




Monday Oct 08, 2007

SAMBA Performance on Sun Fire X4500 (Copying Big Files from 4 CIFS Clients to SAMBA Server)

This is NOT a comprehensive study of SAMBA performance on a Sun Fire X4500, but is a couple of tests I ran because someone wanted some bandwidth numbers.

Plan

See what performance I can achieve copying large files from four windows servers to a Sun Fire X4500 sharing a ZFS file system via SAMBA.

Compare RAID-Z and RAID-Z2. 

Equipment

SAMBA Server 

  • Sun Fire X4500
  • 16 GB Memory
  • 48 x 500 GB SATA-II disks
  • Solaris 10 8/07
  • SAMBA 3.0.25a (as shipped with Solaris)
  • 2 GBE network connections

CIFS Clients

  • 1 x Sun Fire V40z running Microsoft Windows Server 2003 Enterprise X64 Edition SP2
  • 1 x Sun Fire V40z running Microsoft Windows Server 2003 R2 SP2 (32 bit)
  • 1 x Sun Fire X4600 running Microsoft Windows Server 2003 R2 SP2 (32 bit)
  • 1 x Sun Fire X4100 running Microsoft Windows Server 2003 R2 SP2 (32 bit)
These are the OS releases that were installed already. These systems are usually used to test Symantec Enterprise Vault and they all normally have roles within that test environment.

Sun Fire X4500 Configuration

No tuning, everything with out of the box settings.

Tested RAID-Z and RAID-Z2. In both cases created one storage pool, as below.

RAID-Z Configuration

zpool create -f sambapool  \\
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \\
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \\
raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \\
raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \\
raidz c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \\
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0

RAID-Z2 Configuration

zpool create -f sambapool  \\
raidz2 c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz2 c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \\
raidz2 c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz2 c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
raidz2 c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \\
raidz2 c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \\
raidz2 c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \\
raidz2 c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \\
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0

Comment on Storage Pool Configurations

The Sun Fire X4500 has six controllers, each of which support 8 disks (if you want a  picture take a look at this).

In both of the above cases the hot spares are staggered across the controllers to make all the stripes in the pools the same length so that the workload is spread evenly across all the controllers.

SAMBA Configuration (Same in both cases)

Made four directories in the ZFS storage pool and shared each directory via SAMBA.

Below is the smb.conf file I used.

[global]
        workgroup = X4500
        server string = Samba %v
        security = SHARE
[4600e]
        comment = ZFS on X4500
        path = /sambapool/4600e
        force user = root
        force group = root
        read only = No
        guest ok = Yes
[4100b]
        comment = ZFS on X4500
        path = /sambapool/4100b
        force user = root
        force group = root
        read only = No
        guest ok = Yes
[v40c]
        comment = ZFS on X4500
        path = /sambapool/v40c
        force user = root
        force group = root
        read only = No
        guest ok = Yes
[v40b]
        comment = ZFS on X4500
        path = /sambapool/v40b
        force user = root
        force group = root
        read only = No
        guest ok = Yes

Two windows servers access the Sun Fire X4500 via one network interface (e1000g0) and two via the other configured interface (e1000g2). Note that the Sun Fire X4500 has four GBE ports, so two are unused in this configuration

Test Data Used

33 GB of data was created on each Windows Server comprising 24 x 1.4 GB files. These files are test data that I use for some archiving tests. They are ZIP files each holding 50,000 text files with unique content i.e. they ain't full of zeros!

The Test

Copy the test data from all of the windows servers simultaneously to the SAMBA shares (drag and drop with Windows Explorer)

The Results

In both the RAID-Z and RAID-Z2 cases:

  • The network interfaces were reading data off the network at ~90 MBytes/sec for the duration of the test (measured with nicstat).
  • The test completed in the same amount of wall time in both cases (no surprise considering the above was the same)

In the RAID-Z case the internal transfer rate to disks in the Sun Fire X4500 averaged around 200 MB/second and 6000 IOPs (see charts below). CPU utilization peaked at 48%

RAID-Z Data Rate

RAID-Z IOPS


Looking at an individual disk, c0t0d0:

RAID-Z c0t0d0 Data Rate

RAID-Z c0t0d0 IOPS

In the RAID-Z2 case the internal transfer rate to disks in the Sun Fire X4500 averaged around 280 MB/second and 6800 IOPs (see charts below). CPU utilization peaked at 54%.

Note that the collector was started a little late for this run, hence no ramp up. 

RAID-Z2 Data Rate

RAID-Z2 IOPS

Looking at a single disk, c0t0d0, again:

RAID-Z2 c0t0d0 Data Rate

RAID-Z2 c0t0d0 IOPS

Conclusion

With this workload the Sun Fire X4500 has taken the extra work involved in doing RAID-Z2 Vs RAID-Z in its stride. We see elevated levels of  activity on all the disks in the RAID-Z2 case, and under heavier workloads there will inevitably be a point where RAID-Z & RAID-Z2 performance will diverge.

Saturday Sep 01, 2007

New SAMBA focused blog on blogs.sun.com

Jiri Sasek, one of the Engineers who maintains the version of SAMBA that ships with Solaris, has just started blogging. His first entry if all about how to set up a SAMBA Server as a client in an Active Directory Environment. Thanks Jiri!!
About

Tim Thomas

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today