Getting sense of ephemeral store behavior under OpenSolaris (UFS,ZFS)

To get impression of ephemeral store behavior under OpenSolaris (UFS,ZFS) I use bonnie++, compiled from source with debug output. Pre-compiled binary can be get from [www.sunfreeware.com SunFreeware]

This entry is part of 'OpenSolaris on Amazon EC2' workshop

I will skip file based tests and use large file pool test only with size configuration more then 2x of instance memory, because main purpose of test is evaluate "Amazon EC2 documented ephemeral storage feature fist access can be significantly slower ".

Disclaimer

This measurements are done for purpose to get impression of possible bottlenecks, not with any intend to perform deep and log durations performance test.

Hints

On OpenSolaris Nevada line data drive devices are c0dXpX, on OpenSolaris Indiana line data drive devices are c4dXpX

Conclusion

Synthetic test prove then ephemeral storage feature fist access can be significantly slower can be real isuse, we need to take it in account for example wen we will try to backup instance data on OS shutdown.

Version  1.03                  ------Sequential Output------          --Sequential Input-       --Random-
                                     -Per Chr-   --Block--     -Rewrite-   -Per Chr- --Block--      --Seeks--
Machine              Size      K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
1st  ZFSonEC2s 5000M     22925 40   46275  27   27848   12   42671  92 74019    5     296.0 0
2nd ZFSonEC2s 5000M     31250 53   47851  26   32608   14   42355  91 73182   5      272.6 0
3rd ZFSonEC2s 5000M     33538 58   47887  27   32703    14  43116  91 72836    5      287.7 0

Can we do more ? YES

Suggestion are welcome, like:

  • Interesting will be to get sense if ZFS feature copy-on-write can affect real user in EC2 env ?
  • Can we make scrip which will auto import data pool on reboot (ramdisk is in Dom0 and is not updated) ?
  • Can we on Indiana line use ZFS compression to minimalize space used for example by tret based logs ?
  • How to best tune ZFS for Amazon EC2 usage:
  • >
    • Create one smaller pool and pre-write is with zeros, rest of disk spaces use as slower storage ?
    • Tune ZFS memory cache size for each EC2 instance size ?

You can ask you questions at: Help us tune ZFS for usage on 'OpenSolaris on Amazon EC2' project

m1.small disk perf on UFS (1 disk)

m1.small provide to 1x 150GB of Moderate ephemeral storage. Often is used in Linux boxes as plain ext3.

fdisk -B /dev/rdsk/c0d1p0

echo y| newfs /dev/rdsk/c0d1s0

mount -F ufs /dev/dsk/c0d1s0 /mnt
df -h | grep mnt
/dev/dsk/c0d1s0        147G   1.7G   144G     2%    /mnt

/usr/bin/time bonnie++ -s 5000 -x 3 -u 0 -n0 -d /mnt -m UFSonEC2s 

Write speed on first run was significantly slower, its Amazon AWS documented ephemeral storage feature.

Best results are:

Version  1.03              ------Sequential Output------          --Sequential Input-       --Random-
                                 -Per Chr-   --Block--     -Rewrite-   -Per Chr- --Block--      --Seeks--
Machine         Size      K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
UFSonEC2s  5000M     31170 98   54442  75   13512   12  34410  54  71023    32   350.2 1

m1.small disk perf on ZFS (1 disk)

m1.small provide to 1x 150GB of Moderate ephemeral storage. Often is used in Linux boxes as plain ext3.

zpool create mypool c0d1p0

zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
mypool   149G   111K   149G     0%  ONLINE  -

zfs create mypool/zfs

zfs set mountpoint=/zfs mypool/zfs

df -h | grep /zfs
mypool/zfs             146G    18K  146G     1%    /zfs

/usr/bin/time bonnie++ -s 5000 -x 3 -u 0 -n0 -d /mnt -m ZFSonEC2s 

Write speed on first run was upto significantly  slower, its Amazon AWS documented ephemeral storage feature.

Best results are:

Version  1.03              ------Sequential Output------          --Sequential Input-       --Random-
                                 -Per Chr-   --Block--     -Rewrite-   -Per Chr- --Block--      --Seeks--
Machine         Size      K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ZFSonEC2s 5000M     33538 58   47887  27   32703    14  43116  91 72836    5      287.7 0

m1.large disk perf on ZFS mirror (2 disks)

m1.large provide to 2x 420GB of Hi-Performance ephemeral storage. Often is used in Linux boxes as mirror.

zpool create mypool mirror c0d1p0 c0d2p0

zpool list
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
mypool   416G   111K   416G     0%  ONLINE  -

zfs create mypool/zfs

zfs set mountpoint=/zfs mypool/zfs

df -h | grep /zfs
mypool/zfs             410G    18K   410G     1%    /zfs

/usr/bin/time bonnie++ -s 40000 -x 3 -u 0 -n0 -d /zfs -m zfsonEC2x 

Write speed on first run was significantly slower, its Amazon AWS documented ephemeral storage feature.

Best results are:

Version  1.03              ------Sequential Output------          --Sequential Input-       --Random-
                                 -Per Chr-   --Block--     -Rewrite-   -Per Chr- --Block--      --Seeks--
Machine         Size      K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ZFSonEC2x  40000M    68032  71   68953 23    34876 15  73828 92     80325 14     374 2

m1.xlarge disk perf on ZFS raidz (4 disks)

m1.xlarge provide to 4x 420GB of Hi-Performance ephemeral storage. Often is used in Linux boxes as RAID.

zpool create mypool c0d1p0 c0d2p0 c0d3p0 c0d4p0

list
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
mypool  1.62T   114K  1.62T     0%  ONLINE  -

df -h | grep /zfs
mypool/zfs             1.6T    18K   1.6T     1%    /zfs

/usr/bin/time bonnie++ -s 80000 -x 3 -u 0 -n0 -d /zfs -m zfsonEC2xL 

Write speed on first run was significantly slower, its Amazon AWS documented ephemeral storage feature.

Version  1.03              ------Sequential Output------          --Sequential Input-       --Random-
                                 -Per Chr-   --Block--     -Rewrite-   -Per Chr- --Block--      --Seeks--
Machine         Size      K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ZFSonEC2xL 80000M    93203  97 182347 67    88746 45  78486 98     254752 54    557.7 3
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Hands-on experience with Virtualization

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today