Friday Dec 18, 2009

More ZFS Goodness: The OpenSolaris Build Machine

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Apart from my usual LDAP'ing, I also -try to- help the opensolaris team with anything I can.

Lately, I've helped build their new x64 build rig, for which I carefully selected the best components out there while trying to keep the overall box budget on a leash. It came out at about $5k. Not on the cheap side, but cheaper than most machines in most data centers.

The components:

  • 2 Intel Xeon E5220 HyperThreaded QuadCores@2.27GHz. 16 cpus in solaris
  • 2 32GB Intel X25 SSD
  • 2 2TB WD drives
  • 24GB ECC DDR2

I felt compelled to follow up my previous post about making the most out your SSD because some people commented that non mirrored pools were evil. Well, here's how this is set up this time: in order to avoid using either of the relatively small SSDs for the system, I have partitioned the big 2TB drives with exactly the same layout, one 100GB partition for the system, the rest of the disk is going to be holding our data. This leaves our SSD available for the ZIL and the L2ARC. But thinking about it, the ZIL is never going to take up the entire 32GB SSD. So I partitioned one of the SSDs with a 3GB slice for the ZIL and the rest for L2ARC.

The result is a system with 24GB of RAM for the Level 1 ZFS cache (ARC) and 57GB for L2ARC in combination with a 3GB ZIL. So we know it will be fast. But the icing on the cache ... the cake sorry, is that the rpool is mirrored. And so is the data pool.

Here's how it looks: 

admin@factory:~$ zpool status
  pool: data
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        ONLINE       0     0     0
      c5d1p2    ONLINE       0     0     0
      c6d1p2    ONLINE       0     0     0
    logs
      c6d0p1    ONLINE       0     0     0
    cache
      c5d0p1    ONLINE       0     0     0
      c6d0p2    ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        c5d1s0  ONLINE       0     0     0
        c6d1p1  ONLINE       0     0     0

errors: No known data errors
admin@factory:~$

 This is a really good example of how to setup a real-life machine designed to be robust and fast without compromise. This rig achieves performance on par with $40k+ servers. And THAT is why ZFS is so compelling.

Monday Dec 14, 2009

Make The Most Of Your SSD With ZFS

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

 If you're anything like me, you're lucky if you have a single SSD in your workstation or server. I had a dilemma in the past, I couldn't quite find a way to make the most of my one flash drive: I had to chose between making it a ZIL or use it as L2ARC. I dreaded having to make a definitive choice for one or the other. When I installed my workstation with OpenSolaris 2009.06, I had an idea in mind, so I installed the system on the SSD in a small partition (10GB) and left the rest of the drive unallocated if you catch my drift...

Bird's Eye View

Simple! Just partition the SSD to be able to use it as both L2ARC and ZIL in whatever proportions you think is going to suit  your needs. Note however that the IOs are shared between your partitions on the same drive. From my testing though, I can tell you that with this setupu you're still coming out on top in most situations.

The Meat

It's all pretty simple really, when you install solaris, you have a choice of installing on "whole disk" or to use the tool to make a smaller custom partition. I cut out a 36GB partition which allows ample room for the system and swap. The rest of my 64GB SSD is left unallocated at install time, we'll take care of everything later.

The second disk in my system is a 300GB 10,000 rpm SATA drive which, being fast but small, I wanted to leave whole for my data pool (keep in mind that the rpool is a little different than your regular pool, so make sure to treat it accordingly). That is why I decided to compromise and use some of the SSD space for the system. You don't have, you could partition your spindle and have the system on there.

Now, that you have opensolaris up and running, install GParted to be able to edit your disks partitions. You can either use the opensolaris package manager or

pfexec pkg install SUNWGParted

It's all downhill from here. Open GParted. If you just installed it, you will need to log out and back in to see in the GNnome menu. It will be in Applications->System tools->GParted Partition Editor

Select your flash drive and carve out a 2GB partition for your ZIL and assign the remaining space for L2ARC. Apply the changes and keep the window open.


Note the two devices path in /dev/dsk because that's what we'll use to add these two SSD partitions as performance enhancing tools in our existing pool.

arnaud@ioexception:/data/dsee7.0/instances$ pfexec zpool add data log /dev/dsk/c9d0p2 cache /dev/dsk/c9d0p3

Let's check how our pool looks now...

arnaud@ioexception:/data/dsee7.0/instances$ zpool status data
  pool: data
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        ONLINE       0     0     0
      c8d0      ONLINE       0     0     0
    logs
      c9d0p2    ONLINE       0     0     0
    cache
      c9d0p3    ONLINE       0     0     0

errors: No known data errors

Et voila!

You've got the best of both worlds, making the absolute most of whatever little hardware you had at your disposal!

Enjoy!

Add And Remove ZILs Live!

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Rationale

Ever been playing with separate logs (separate ZIL, logzilla, etc...) and had to rebuild the pool everytime you wanted to yank the slog off ?

Not so anymore! The zil can now be added and removed as you like! Truly fantastic to tinker with it and observe its actual impact on performance. A quick walk through one of the most painless migrations of all times.

Bird's Eye View

As ZILs get more and more exposure to production environments, technical staff is getting to experiment more and more to be able to make recommendations and the new removal feature adds a lot more flexibility to try creative combinations.

The Meat

Let's suppose you have a pool with a separate log:

arnaud@ioexception:/data/dsee7.0/instances$ zpool status data
  pool: data
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
    still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
    pool will no longer be accessible on older software versions.
 scrub: none requested
config:

    NAME                      STATE     READ WRITE CKSUM
    data                      ONLINE       0     0     0
      c8d0                    ONLINE       0     0     0
    logs
      /dev/ramdisk/zil-drive  ONLINE       0     0     0

errors: No known data errors

If you try to remove the log by removing the actual separate log device, you'll get the following error:

arnaud@ioexception:/data/dsee7.0/instances$ pfexec ramdiskadm -d zil-drive
ramdiskadm: couldn't delete ramdisk "zil-drive": Device busy

If you now try to use the zpool remove command, you will also hit a wall:


arnaud@ioexception:/data/dsee7.0/instances$ zpool remove data log /dev/ramdisk/zil-drive
cannot remove log: no such device in pool
cannot remove /dev/ramdisk/zil-drive: pool must be upgrade to support log removal

So let's just follow up on the suggestion and upgrade the pool: 



arnaud@ioexception:/data/dsee7.0/instances$ pfexec zpool upgrade data
This system is currently running ZFS pool version 22.

Successfully upgraded 'data' from version 14 to version 22

arnaud@ioexception:/data/dsee7.0/instances$ zpool status data
  pool: data
 state: ONLINE
 scrub: none requested
config:

    NAME                      STATE     READ WRITE CKSUM
    data                      ONLINE       0     0     0
      c8d0                    ONLINE       0     0     0
    logs
      /dev/ramdisk/zil-drive  ONLINE       0     0     0

errors: No known data errors

Wow, that's quick, easy and smooth. Gotta love migrations of that sort. Let's now try to remove our separate log:


arnaud@ioexception:/data/dsee7.0/instances$ pfexec zpool remove data log /dev/ramdisk/zil-drive
cannot remove log: no such device in pool
arnaud@ioexception:/data/dsee7.0/instances$ zpool status data  pool: data
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        ONLINE       0     0     0
      c8d0      ONLINE       0     0     0

errors: No known data errors

So even though it barfed an error message, my separate log has been removed from my pool and I will be able to decommission my device.

arnaud@ioexception:/data/dsee7.0/instances$ pfexec ramdiskadm -d zil-drive
arnaud@ioexception:/data/dsee7.0/instances$

And I can now create a new device and do some more testing. Props to the ZFS for the ever improving level of service this file system brings to the table!

Friday Oct 02, 2009

Note To Self: Things To Do On A Vanilla System

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

I just thought I'd make a note of the common things I do and funny enough, I think this blog might be the closest thing I have from a sticky note / persistent backup ... so here goes:

 PATH=/usr/bin:/usr/sbin:/usr/gnu/bin:$PATH
# enable power management
pfexec echo "S3-support    enable" >> /etc/power.conf
pfexec pmconfig
pfexec svcadm restart hal

# disable access time update on rpool to minimize disk writes
pfexec zfs set atime=off rpool

# get pkgutil to install community software
pfexec pkgadd -d http://blastwave.network.com/csw/pkgutil_`uname  -p`.pkg

# Add repositories to the package manager

pfexec pkg set-publisher -O http://pkg.opensolaris.org/dev/ dev
pfexec pkg set-publisher -O http://pkg.opensolaris.org/contrib/ contrib
pfexec pkg set-publisher -O http://blastwave.network.com:10000/ blastwave
pfexec pkg set-publisher -O http://pkg.sunfreeware.com:9000/ sunfreeware


# download and install the flash plug-in for firefox
wget http://fpdownload.macromedia.com/get/flashplayer/current/flash_player_10_solaris_x86.tar.bz2 -O libfp.tar.bz2 --no-check-certificate
bunzip2 libfp.tar.bz2
tar xf libfp.tar
pfexec mv flash_player\*/libflashplayer.so /usr/lib/firefox/plugins
rm libfp.tar
rmdir flash_player\*

# get perfbar
wget http://blogs.sun.com/partnertech/resource/tools/perfbar.i386 -O perfbar --no-check-certificate
chmod 755 perfbar
nohup ./perfbar &

# configure coreadm
coreadm -g /var/cores/%t-%f -e global

Quad Monitor With Rotation: Where There Is A Will, There Is A Way

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

Talking with a friend recently, he told me about his miserable experience trying to get his workstation to work with four monitors.

Now, I was surprised at first because there are lots (ok, maybe not lots, but a sizeable number) of people with quad-head workstations out there, so obviously that seems rather doable. The trick in his case seemed to be heterogeneity: 2 different dual-head cards, and 4 different monitors of different brands and sizes. Additionally, he wanted one of his widescreens tilted in portrait mode for his coding. Nice for browsing as well, but he wanted to be able to have a tall IDE to see more code at once without the need to scroll.

It took me a while just to get the equipment but to find some spare time to this as well. I ended up with the following:

  1. a desktop that would lend itself to the experiment
  2. 4 dual head videos cards to test combinations
    1. nVidia GTX 280
    2. nVidia Quadro FX 380
    3. nVidia GeForce 9600 GT
    4. nVidia GTS 250
  3. 4 monitors
    1. Sun 24.1"
    2. Dell 22"
    3. Acer 24.3"
    4. Dell 20"
  4. a free Saturday (that was actually the most difficult component to find)

To cut short, the result is ... rolling drum ... it _can_ work once you know what to do and what not to. Here is the final result:

So how do we make that work? Well, first thing is NOT to desperately cling to TwinView. You have to let go of that, fall back on good ol' XineRama which does a fine job anyway.

As I said in my previous post, rotating the monitor is only a matter of adding Option "Rotate" "left" in the relevant screen section.

For all the X options explained, I found this quite useful. Dig in there.

What you want to be careful about:

  • if at first both cards are not recognized, worry not. Go to a terminal and issue the following command:

pfexec nvidia-xconfig -a

This will force the nvidia config utility to look across all cards.

Note that if this still doesn't work, issue:

pfexec scanpci

and write down the PCI id for each card. It is the first number right after the pci bus 0x002. In this example, this would translate into

BusID "PCI:2:0:0"

in the device section in xorg.conf

  • look at your /var/log/Xorg.0.log for errors
    • you will see something like

(II) LoadModule: "xtsol"
(WW) Warning, couldn't open module xtsol
(II) UnloadModule: "xtsol"
(II) Failed to load module "xtsol" (module does not exist, 0)

 Don't worry, that's a trusted solaris extension that is hardcoded to be loaded by X even when it's not a trusted solaris OS running, this has yet to be fixed. 

  •  make sure to enable Composite
  • make sure to enable GLX with composite
  • make sure to enable RandRRotation
  • Check /var/adm/messages for IRQ collisions which could result in some funky discrepancies. If you find any, tweak your BIOS to force each PCI slot to a distinct IRQ. The message would look similar to:

unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels

All that said, here is an example of xorg.conf with a single monitor tilted, and everything working pretty well considering that nothing is matched. It does work but doesn't come for free as you can see. There is one drawback however, I have not been able to make Compiz work because apparently the cards would have to have an SLI link between them, but I haven't confirmed that for sure. That's it for today folks!

OpenSolaris 2010.02 on EeePC 1000: Out-Of-The-Box!!!

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

As usual, I try to give as much away in my titles as I can. This one is no different: it just works....


With 2009.06, you needed to build your own drivers for ethernet and wifi. Pretty much a non starter for 99% of users, understandably so: when it just works for Linux and Windows, why sweat it on OpenSolaris ?

Now that dilemma is behind us: I installed an early access of 2010.02 (OpenSolaris b124) and when the installation was done, everything worked: a whole new  experience for me on OpenSolaris. I almost EXPECT to have to fiddle with a driver, a config file, an SMF service that doesn't start, ..., something.

In this case: nothing! Simultaneously gratifying and almost disappointing. I mean, even on my desktop OpenSolaris required some elbow grease to work  the way I wanted, but in this case, the coveted prize of a functional system would be handed to me without even the hint of a fight ? ... unusual, to say the least.

And that's good. I used to say that Solaris is the certainly best server OS and just as certainly the worst desktop OS, but this one shot has me wondering... maybe the Sun engineers have covered some of the ground that separates OpenSolaris from Linux. Granted, there's still ways to go! Yes the embedded 1.3 Mega Pixels webcam works but the quality of the picture is perfectible and I don't think it is the hardware... to be fair, Sun has to write their own drivers for everything so I'm even surprised it worked at all, so that pretty good!

Now there is on rather big bummer though...it does suspend but doesn't resume. Pretty big issue for a laptop which is -because of its form factor- bound to be used on the go. If I can make it work, I will post here. If you have had success make resume work, drop me a line!



Lenovo W700ds dual monitor laptop: works! Another 2010.02 success

<script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try { var pageTracker = _gat._getTracker("UA-12162483-1"); pageTracker._trackPageview(); } catch(err) {}</script>

OpenSolaris 2010.02 early access build 124 is really faring pretty well so far. It isn't free of issues, granted, but at the same time, it has improved leaps and bounds on laptop support, especially for netbooks, thanks to a passionate and dedicated team writing up a bunch of device drivers for wifi and network cards found in these little laptops.

Today, I installed build 124 on a Lenovo W700ds.

You probably have never heard of that beast because they probably only sold half a dozen of them, one of which landed on my desk yesterday. The main reason for this success is probably that it weighs a ton (11 lbs or 5 Kgs!!!) due in part to its main 17" monitor, doubled by a netbook-like 10" monitor that slides out from behind the main one.... here are the specs. Notice they call it "portable power". Trasnportable would be more accurate. After using this laptop for about an hour now (I'm writing this post on it), I do have to say that it is quite fantastically comfortable, just about as much a desktop would be...not really surprising if you consider it has a full size keyboard + numeric keypad.

 So, OpenSolaris installs without a glitch, once again the installer just does its job without whining. If you run the device driver utility it will notify you that two devices do not have a driver for solaris, one being the integrated bluetooth card and the other being the fingerprint reader. Not a big deal. Once OpenSolaris is installed, it will boot in Gnome just as on any other machine, but what you really want is the second monitor to work... and there's a trick to that.

 First, the second monitor won't be recognized if you don't pull it all the way out at boot time. Took me a while to figure this one out. To save some mW, the Lenovo folks don't power it unless it's out and that makes it undetectable at first.

Second, once recognized by X, it will actually display sideways. This "companion" display is actually is 16:9 10" netbook display tilted right so that it's width resolution (1280x768) almost matches the height resolution of the main display (1920x1200). So all we have to do is to tilt it "left" to compensate for the hardware arrangement. To do so, simply enable the Rotate and Resize option on the graphics card and then tell X to rotate the appropriate screen left. Here's how:

Section "Monitor"

    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "SlideOut"
    VendorName     "Lenovo"
    ModelName      "LEN 2nd Display"
    HorizSync       30.0 - 75.0
    VertRefresh     60.0
    Option         "DPMS"
    Option         "Rotate" "left"
EndSection

Section "Device"
    Identifier     "Device1"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 2700M"
    BusID          "PCI:1:0:0"
    Screen          1
    Option       "RandRRotation"    "on"
EndSection

 Note that TwinView must be disabled because twinview aggregates both display into a single block. Rotation with twinview on will result in rotate both displays. So you need to make them two X displays and enable xinerama.

here is the final xorg.conf in case you're interested...

Additional notes:

Suspend/Resume works great with this laptop -most of the time- however, it seems that sometimes, you will lose the second display upon resume, I'm not sure why.

About

Directory Services Tutorials, Utilities, Tips and Tricks

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today