SAMBA Performance on Sun Fire X4500 (Copying Big Files from 4 CIFS Clients to SAMBA Server)

This is NOT a comprehensive study of SAMBA performance on a Sun Fire X4500, but is a couple of tests I ran because someone wanted some bandwidth numbers.

Plan

See what performance I can achieve copying large files from four windows servers to a Sun Fire X4500 sharing a ZFS file system via SAMBA.

Compare RAID-Z and RAID-Z2. 

Equipment

SAMBA Server 

  • Sun Fire X4500
  • 16 GB Memory
  • 48 x 500 GB SATA-II disks
  • Solaris 10 8/07
  • SAMBA 3.0.25a (as shipped with Solaris)
  • 2 GBE network connections

CIFS Clients

  • 1 x Sun Fire V40z running Microsoft Windows Server 2003 Enterprise X64 Edition SP2
  • 1 x Sun Fire V40z running Microsoft Windows Server 2003 R2 SP2 (32 bit)
  • 1 x Sun Fire X4600 running Microsoft Windows Server 2003 R2 SP2 (32 bit)
  • 1 x Sun Fire X4100 running Microsoft Windows Server 2003 R2 SP2 (32 bit)
These are the OS releases that were installed already. These systems are usually used to test Symantec Enterprise Vault and they all normally have roles within that test environment.

Sun Fire X4500 Configuration

No tuning, everything with out of the box settings.

Tested RAID-Z and RAID-Z2. In both cases created one storage pool, as below.

RAID-Z Configuration

zpool create -f sambapool  \\
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \\
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \\
raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \\
raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \\
raidz c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \\
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0

RAID-Z2 Configuration

zpool create -f sambapool  \\
raidz2 c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz2 c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \\
raidz2 c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz2 c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
raidz2 c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \\
raidz2 c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \\
raidz2 c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \\
raidz2 c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \\
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0

Comment on Storage Pool Configurations

The Sun Fire X4500 has six controllers, each of which support 8 disks (if you want a  picture take a look at this).

In both of the above cases the hot spares are staggered across the controllers to make all the stripes in the pools the same length so that the workload is spread evenly across all the controllers.

SAMBA Configuration (Same in both cases)

Made four directories in the ZFS storage pool and shared each directory via SAMBA.

Below is the smb.conf file I used.

[global]
        workgroup = X4500
        server string = Samba %v
        security = SHARE
[4600e]
        comment = ZFS on X4500
        path = /sambapool/4600e
        force user = root
        force group = root
        read only = No
        guest ok = Yes
[4100b]
        comment = ZFS on X4500
        path = /sambapool/4100b
        force user = root
        force group = root
        read only = No
        guest ok = Yes
[v40c]
        comment = ZFS on X4500
        path = /sambapool/v40c
        force user = root
        force group = root
        read only = No
        guest ok = Yes
[v40b]
        comment = ZFS on X4500
        path = /sambapool/v40b
        force user = root
        force group = root
        read only = No
        guest ok = Yes

Two windows servers access the Sun Fire X4500 via one network interface (e1000g0) and two via the other configured interface (e1000g2). Note that the Sun Fire X4500 has four GBE ports, so two are unused in this configuration

Test Data Used

33 GB of data was created on each Windows Server comprising 24 x 1.4 GB files. These files are test data that I use for some archiving tests. They are ZIP files each holding 50,000 text files with unique content i.e. they ain't full of zeros!

The Test

Copy the test data from all of the windows servers simultaneously to the SAMBA shares (drag and drop with Windows Explorer)

The Results

In both the RAID-Z and RAID-Z2 cases:

  • The network interfaces were reading data off the network at ~90 MBytes/sec for the duration of the test (measured with nicstat).
  • The test completed in the same amount of wall time in both cases (no surprise considering the above was the same)

In the RAID-Z case the internal transfer rate to disks in the Sun Fire X4500 averaged around 200 MB/second and 6000 IOPs (see charts below). CPU utilization peaked at 48%

RAID-Z Data Rate

RAID-Z IOPS


Looking at an individual disk, c0t0d0:

RAID-Z c0t0d0 Data Rate

RAID-Z c0t0d0 IOPS

In the RAID-Z2 case the internal transfer rate to disks in the Sun Fire X4500 averaged around 280 MB/second and 6800 IOPs (see charts below). CPU utilization peaked at 54%.

Note that the collector was started a little late for this run, hence no ramp up. 

RAID-Z2 Data Rate

RAID-Z2 IOPS

Looking at a single disk, c0t0d0, again:

RAID-Z2 c0t0d0 Data Rate

RAID-Z2 c0t0d0 IOPS

Conclusion

With this workload the Sun Fire X4500 has taken the extra work involved in doing RAID-Z2 Vs RAID-Z in its stride. We see elevated levels of  activity on all the disks in the RAID-Z2 case, and under heavier workloads there will inevitably be a point where RAID-Z & RAID-Z2 performance will diverge.

Comments:

I have yet to see any performance data which suggests that a controller bottleneck exists for the X4500. For SAS/SATA, the disks do not generally share a path. This is unlike IDE, parallel SCSI, or Fibre Channel paths where it is very common to see multiple devices sharing the bandwidth and arbitration characteristics of a path. In those cases, we can measure resource contention on the path itself. Consider the 200 MBytes/s case in your data, all of the controllers are loafing.

Posted by Richard Elling on October 11, 2007 at 01:22 PM BST #

Would it be possible to perform a similar test using NFSv3?

Posted by Missing Droid on October 23, 2007 at 08:38 AM BST #

No, I have moved onto other projects now and don't have the equipment anymore....sorry.

Posted by Tim Thomas on October 23, 2007 at 08:40 AM BST #

Post a Comment:
Comments are closed for this entry.
About

Tim Thomas

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today