Tuesday Feb 06, 2007

Getting Solaris x86 Graphics, NIC and Audio working on the ECS GeForce 6100SM-M

There was a recent sale at Fry's on CPU+ mobo combo. This was for an Athlon 64 x2 3800+ 65W AA processor retail kit with an ECS GeForce6100SM-M motherboard - all for $139 before tax. The motherboard is based on the nVidia MCP61 (nForce 405) chipset with socket 939, PCI-express x16 and x1 slots, DDR2 800 memory slots, the standard I/O for disks and floppy and USB, onboard nVidia GeForce 6100 graphics, built-in nVidia 10/100 Fast Ethernet and High-Def Audio with a ALC 660 Codec.

Getting Solaris to run well on this system wasn't the easiest thing. While GeForce 6100 is a supported graphics chip, the version on this board wasn't recognized by the bundled 'nv' driver which we collaborate closely with nVidia to get done with the most recently drop dated January 2007, just weeks ago. The on-board NIC was a strange device with what looked like a 10/100/1000 capable PHY part but a chip (pci10de,3ef) only capable of 10/100 Fast Ethernet speeds. And the audio was another hybrid of sorts that used the MCP61 HD Audio controller coupled with a Realtek ALC660 codec. The controller seems to function to the Intel HD Audio spec and therefore similar to all the other nVidia Azalia (codename for HD Audio) controller. But the codec was a cheaper version of 5.1 audio, as opposed to the standard 7.1 Surround Audio for most HD Audio Codecs (such as the ALC880, 882, 885, etc.). And as expected, the back audio I/O ports included only a single column of 3 jacks for Line In, Mic, and Line Out.

Using the VESA graphics driver with 16 bits.

The GeForce 6100 graphics wasn't suffering the usual errors and exit issues that'd I'd expect from a unsupported card. Instead, it was starting and the system thought it was starting, but my 20 inch flat panel was complaining that the signal was at a mode unsupported by the monitor and the monitor would blank at that point. To get the Graphics working, I examined the /var/log/Xorg.0.log file and found that the actual startup of graphics fell through the 'nv' driver due to errors caused by missing modules for GLX and, instead, loaded the VESA module. Even though the BIOS was set to share 64MB of memory with Graphics, I couldn't get the VESA driver to display the native 1600x1200 resolution of my 20 inch flat panel. The command line options are in the Xorg.0.log file but I had assumed everything was kosher.

I managed to keep the system in command-line mode and manually fired up Xorg by typing: /usr/X11/bin/Xorg, and suddenly, X came up in a standard grey, houndstooth style screen. But if I executed /usr/X11/bin/X (which is a script that sets up the Xserver that then calls Xorg) then it fails. I put a "set -x" line into the /usr/X11/bin/X script and observed how it started and discovered quite a bit of initial checking to discover the default bit-depth of the screen. Without any environment settings, the system defaults to 24-bits per pixel and formats a command-line with a number of options that includes the bit depth. This is then passed to Xorg. so the difference between calling Xorg with no args and X was the set of options. Using the process of elimination, I then determined that without -depth 24, the X script also can start the Xorg server and does so using a default 16 bpp for 1600x1200 resolution. I could actually get the VESA driver to do 24 bpp with 1280x1024, but that was far more blurry and not the native resolution. So I decided to correct the /usr/X11/bin/X script and add a line prior to the initialization of the Xserver setting the DefaultDepth to 16bpp. This allowed the GUI to come up in native resolution with 32k colours. This is good enough for most uses, but application popup menus are colored incorrectly when overlapping another window under 16 bpp. It's tolerable and still works, but I'd prefer 24 bpp and 1600x1200 resolution support.

HD Audio modifications

We have some minimalist HD audio support currently shipping with Solaris 10 update 3 and Open Solaris. The driver actually supports a growing list of Codecs from Realtek, Analog Devices and Sigmatel. The architecture of HD audio is different from the old AC'97. Both specs are from Intel, but the newer HD Audio spec isn't designed as a superset of AC'97 features. It is a new device architecture that separates controller from codec. So the MCP61 controller appeared to be much like all the previous nVidia Azalia controllers. In fact, by adding the line: "audiohd "pci10de,3f0" to the /etc/driver_aliases file, and the running "update_drv audiohd" and "devfsadm" then rebooting, the audiohd module in Solaris does load and almost attaches. It errors out however because inside the driver module, there is a codec initialization routine that fails because the codec isn't recognized.

The funny thing about this ALC660 codec is that it has 5.1 channels, as opposed to 7.1 for most HD Audio codecs. It's a short cut that removes some of the audio output pins. I thought perhaps that this was probably very similar to ALC880 and ACL882 codecs and I could probably tickle the same pins using the same code. Only, there'd be one fewer pair of pins. While the risk of frying the part can exist if you configure it incorrectly in a driver, the chance of that really happening was slim and something I guessed was worth it, if I could even get the system to play just a noise.

Within the codec initialization in the Solaris audiohd.c code are a number of switch/case blocks that do the work. I started by modifying the audiohd_impl.h header file and added a new AUDIOHD_VID_ALC660 entry which corresponds to a pci10ec,0660 device ID. This actually is attached to the PCI-express bus. Next, I opened the audiohd.c file and added entries in a few dozen places wherever I saw ALC880/882. Rebuilding the 32- and 64-bit drivers, and adding them to the /kernel/drv and /kernel/drv/amd64 directories, I rebooted then enabled the audio to play.

Open source nVidia Ethernet driver

The later Nevada builds and Solaris 10 update 3 all support the nVidia onboard GigE device (nge), but the MCP61 networking chip has an unrecognized device ID (pci10de,3ef). I tried to add this also to the /etc/driver_aliases, and run my update commands and reboot, but while the module loaded, it did not attach to the device and I couldn't get it to plumb. Seaching on the web, I encountered Murayama's Free Solaris NIC drivers and an alpha version of the nfo-2.4.1 driver. It does support a number of nVidia on-board fast ethernet chips, but none with the same ID. I attempted to try and attach it anyway, and the message logs told me that the driver attempted to attach but it failed to find it in the nfo NIC table. I looked at the nfo_gem.c file and found an array declaration for the nfo_nictbl[] that had a list of 15 devices supported so far, then cloned the last entry and added a 16th with the new device ID supporting 64-bit and JUMBO frames. I wasn't sure if that was the case, but I recompiled and copied these back into their respective /kernel/drv and /kernel/drv/amd64 and magically the interface came up when I rebooted and manually did an "ifconfig nfo0 plumb".

The next step was to access the network and bringover about 15 GB of files and sample audio which came over quickly and without any issues. This was to do further testing on the audio driver as well, which continued to play just fine.

All in all, not the most straight forward of installs, and not hands free, but it was relatively painless and largely made easy because of the community and open source.

Friday Jan 05, 2007

Giving USB the boot - install

A sale on 4 GB USB jumpdrives ended yesterday at Fry's. It wasn't a huge savings, but for $49, there was a USB flash drive from PQI that seemed fairly decent and I bought one. I wasn't sure what I was going to do with it, but I was thinking along the lines of using it to burn Solaris Nevada b55 onto so I could carry it with me always and install other systems that support boot from USB. It's big enough afterall.

There's been lots of news from India about the Belenix folks on the Solaris-runnable Flash archive on the 1GB stick. It's very cool and the person -Anil Gulecha- who did the runnable flash stick image has been getting lots of praise. I thought I'd join in, since I leveraged a lot of things he was doing in his usbdump.sh to create a bootable Solaris Nevada b55 image onto that USB drive (on sale!).

Overview of Steps

Basically, the process is fairly straightforward, except the commands aren't always known or easy to remember.

    1. Format the USB drive and create a UFS filesystem on it.
    2. Copy over the files verbatim from the install DVD
    3. Install GRUB to make it bootable
    4. Boot a System to Install with the USB drive

As I discovered, the contents of the install DVD aren't quite amenable to a straightforward installation. There are definite dependencies for example, the path where the DVD install scripts expect the media to be (i.e. /cdrom). And this doesn't make life easy unless you know the workarounds. But as I discovered, the USB drive installation is about 30% faster than standard DVD install when it all works, and it's a lot quieter than a spinning optical drive. It also works when the system doesn't support PXEboot net-install, or the driver for the network isn't by default in the x86.miniroot for the installation OS. The flash drive, being a disk, makes it easy to include additional software like driver source files, the Studio 11 compiler packages, and optional freeware.

Formatting a USB flash drive for Solaris

This was shamelessly reverse engineered from Anil's usbdump.sh script and it seems to work pretty well. For those that don't want to extract out the relevant shell script lines to make their own script, basically, this is what I did to format my jump drive.

    1. stop volfs on your current Solaris 10 or Nevada box by typing the command: # svcadm disable volfs
    2. plug the USB flash drive in, wait several seconds
    3. run the command # rmformat -l to lists the removable media devices and find the Connected device which says something to the order of a Generic USB flash disk, and note the line which says: Logical Node: /dev/rdsk/c3t0d0p0 or something like that. I'll call the /dev/rdsk/c3t0d0s0 the [usbnode] going forward. You'll need to know yours later.
    4. # fdisk -B [usbnode] (where [usbnode] is from previous step)
    5. # fdisk -W - [usbnode] (need the extra hyphen for stdout - note the output
    6. Now, look at output from previous step and find max MB (MAXSIZE) of usable storage. You do this by multiplying bytes/sector (usually 512 at the top) with num sectors at the bottom of output (in my case, a 4 GB drive had roughly 7.9Million sections or so) divide by (1024\*1024) to yield MB (should be 3900 MB or so for a 4GB flash stick.
    7. setup your partition table file (a text file I'll call slices.txt, just plain white space between words and letters where it appears (remove dashed lines - eval {$MAXSIZE - x} expressions and replace with number with no brackets (e.g. 3897MB)

      slices: 0 = 2MB, {$MAXSIZE-3}MB, "wm", "root" :
               1 = 0, 1MB, "wu", "boot" :
               2 = 0, {$MAXSIZE - 1}MB, "wm", "backup"

    8. # rmformat -s ./path/to/slices.txt [usbnode]
    9. now run: # devfsadm; devfsadm -C
    10. Create a UFS on the flash drive by runing: # newfs /dev/rdsk/[usbslice] (slice means replace p0 with s0 in the [usbnode] path)
    11. wait for a minute to write, and now it's done.
    12. You can restart the volfs by running # svcadm enable volfs but you may want to hold if you plan to proceed to the next step immediately.

Loading the Install Media onto Flash drive

For the time it takes to burn a DVD with the Solaris install media, it's almost possible to copy the files onto disk using Solaris' loopback mount filesystem capability. The utility is lofiadm(1M) and is really useful for mounting ISO images already present on the filesystem and saves the hassle of burning media only to insert the finished DVD and remount it. Steps are the following to mount the Solaris install media ISO and copy them to the flash:

    1. # lofiadm -a /absolute/path/to/solarisdvd.iso (this should create a node /dev/lofi/1 which is reported on the command line)
    2. created a directory, e.g. # mkdir /mnt/solmedia and then run # mount -F hsfs -o ro /dev/lofi/1 /mnt/solmedia (you can change directories into /mnt/solmedia and check to see that the media is indeed there.
    3. disable the volfs again by running svcadm disable volfs if you re-enabled it after the prior task of formatting the flash drive.
    4. Insert the USB flash drive then wait a second and run the rmformat -l to list the logical node. You might want to create a directory mount point (e.g. # mkdir /mnt/usbdrive) and then run:
      # mount -o nologging,noatime [usbslice] /mnt/usbdrive
      Remember that the [usbslice] is the same as the logical [usbnode] but we created slice0 on that flash drive so we swat the p0 with s0 (e.g. /dev/dsk/c3t0d0p0 -> /dev/dsk/c3t0d0s0).
    5. now simply change directories to /mnt/solmedia and copy everything over. I've used the command # cd /mnt/solmedia; cp -rP@ .??\* \* /mnt/usbdrive. And once complete, should put all the media onto the USB drive.

Note that with some flash drives, including this PQI, I had data errors. At least if the error occurs, Solaris will usually tell me and then I note the error in the package and try to re-copy it later. What's more insidious is if the drive doesn't report errors initially but then encounters a few errors during the install and fails to install, say some of the Xorg components or fails to run the post-install on the fonts and then your windowing won't be able to come up. More details below.

Booting and Installing with the flash drive

On one IBM/Lenovo laptop we tried, we inserted the jump drive and the system BIOS knew to boot from the USB drive immediately. On an Acer Ferrari 3400, I like others who've reported the same thing, had to go into BIOS and set the boot order to be from a Hard Disk. Not a USB zip, USB floppy, or other USB drive. And on a PCChips V21G system, I pressed F11 to get into the boot menu and then found the Generic USB boot drive under the +Harddisk boot sub-menu.

Once booted, it looks just like the DVD/CDROM install. Only, the install scripts for this media all are expecting the installation content to be mounted under /cdrom. And so the boot will proceed partially, and then exit to a #sh prompt when it can't proceed with the installation. The solution I found was to mount the flash filesystem under /cdrom. But finding the logical [usbslice] to mount is a bit tougher. The rmformat utility isn't in the installers x86.miniroot so that utility isn't available. The next best thing is to look in /dev/usb/ and do a full listing (ls -l mass-storage\*) and observe the path. You should see one or more entries that should point to the physical path in the /devices/ tree. Note the exact path, which can be tedious, but if we back out directories to ../dsk/ and then do a full listing, we should see a similar slice. We want to mount the path for slice0. This usually ends up being /dev/dsk/c1t0d0s0, /dev/dsk/c0t0d0s0 or /dev/dsk/c2t0d0s0. If you aren't sure, just give those 3 entries a try using this command as an example: # mount -o ro /dev/dsk/c1t0d0s0 /cdrom.

To continue with the install, I think folks are recommending /sbin/install-solaris. But since we haven't gotten to the initial questions about hostname, networking, locale, etc., I've been using the /sbin/install-begin script. I hope someone reading this in the install group knows how to kicks this off again, and corrects me if this isn't the preferred way.


I was able to install on a couple of systems, and they booted, but Xorg failed to come up and I think it had something to do with corruption on a couple of font packages and Xorg drivers. Also, on a triple boot laptop, I was also having some issues getting the installer to go into Upgrade mode. It couldn't see the slice for Solaris, but it worked fine for a double-boot and . There's still bugs I guess, and some may be related to data corruption on these flash drives, which seem buggy. But it's not a bad start to get things to boot and come up. So that's progress. The next step is to look at adding some support into the script for install_media=[a local filesystem path] if someone hasn't done it already, and also to see if I can use a USB portable hard drive to see if this eliminates the data corruption errors.

In addition, the current Solaris install is fragile because it hard codes support (and therefore limits it) to /cdrom and nfs mount paths currently. I guess with access to the x86.miniroot for the installer, anyone might potentially add some support for USB drive mounting and installation off of that media with the right amount of clever scripting. That's probably a project requiring a whole weekend though, unless somebody's already done it.

Gotta run and pickup my new 2007 California fishing license and go fishing. Probably not something to hack this weekend.

Wednesday Dec 06, 2006

Swapping drives between Solaris machines

AMD Geode NX and motherboard combo deals

In the constant search for more low-power, home system deals, another processor I've had good success with is the AMD Geode NX. The Geode is a line of x86-compatible 32-bit processors AMD offers for various embedded applications, usually in extremely low-power, space-constrained applications. I'm not sure what the roots are of the Geode line, but at the higher end of the spectrum is the Geode NX and from what folks are saying on the web, it looks like these have roots from the Athlon XP line and might actually be the cream-of-the-crop CPUs that are down-clocked and can run stably at 1.2V (rather than the usual 1.6V) so as to reduce power. I've only seen Geode NX 1750 1.4 GHz cpus sold on combos with motherboards retail but AMD advertises multiple speed versions of the Geode NX, probably for volume OEMs.

About two months ago, there was a sale at Newegg.com on a particular Geode NX/Motherboard combo. The price on the combo was $69.99. This deal was for a PCChips M863G(v7.0) micro-ATX socket-A board with SiS 741GX/964L chipset and integrated graphics, AC'97 audio and SiS900 fast ethernet. The combo included a factory mounted, Geode NX 1750 idling which supposedly idles at 14 Watts going up to 20 Watts. The combo included a generous heat sink and super loud 70x15mm fan that ate 7.3Watts by itself! (Fan and H/S replaced with much quieter CoolerMaster unit running at 7V and 2200 rpm). Rev 7.0 of this board does support a 1.2V core cpu voltage setting that lets the Geode NX achieve its low power status. Newegg seems to be sold out of this combo, but they get more once in a while. (Picture courtesy of Newegg.com).

Fig. 1. PCChips M863G board with bundled AMD Geode NX 1750 CPU from Newegg.COM

Directron.com is also sold-out on a similar combo with a Biostar M7VIG 400 board with all-in-one VIA KM266 Pro chipset. The Biostar board should have been the better board and I actually bought this board first, paying a few bucks more because I knew the chipset better and thought it was well supported. But after buying it, I wasn't fully happy because the board revision and BIOS didn't support the Geode NX's low-voltage mode. So while the cpu could run cooler and with the same clock rate, 1.4V was the lowest voltage it could drop to, so basically, I was sort of running the equivalent of an Athlon XP at 30 - 35 Watts, when the cpu should be able to operate at 14 - 20 Watts. I also had some issues with Xorg on the Unichrome graphics which I was able to work around, but it wasn't a clean install. The Xorg workaround will be explained a little further down.

So with the Biostar-VIA board not being as low-power as I thought, and Newegg having the similar combo with SiS chipset, I didn't hesistate; I bought one. As luck would have it, not 10 days after I received the first board, Newegg had the $10 rebate offer that would save more on the same item, so I had to take advantage of that deal too and get a second combo to score on the rebate. I did this pretty much not having checked if the SiS graphics, network and audio would work - the deal was just too good to pass up.

Sad days; Retirement of two old friends coming soon.

The goal of buying those Geode NX combos was to begin the process of retiring a pair of 8 year old boxes that have served me well over the years. These were proprietary BookPCs I bought back in 1998 with a tiny Super Socket 7 mobo, special 95 Watt power supply, and were short depth. These had the VIA Apollo MVP3 chipset with ECC SDRAM support and I maxed each one out with 512 MB of branded ECC memory and AMD K6-2 450 MHz. These were tough boxes and the weight alone on these small but heavy units was definitely old school manufacturing. They had Davicom 9102 NICs and I only put a disk drive into each, removing optical drive and FDD after the installation for reduced cabling and improved airflow. I've upgraded these boxes over the years. First 6.4GB IDE drives in both, then 40 GB Ultra quiet drives. Today, one runs Linux kernel 2.4 still and the other runs Solaris Nevada. Both have run so long that the power supply and CPU fans have clogged up from dust and seized up and the power supplies (which are very hard to find) have blown. I spent a couple of days back in 2001 relearning basic circuit analysis again and researching power supplies. After a couple of trips to Halted Supply Co. (HSC) near Lawrence Expressway and Central in Santa Clara, I finally got a bag of high frequency switching capacitors in various sizes at about $0.35/each. So for about $1.50 per power supply plus $7 for a new fan, I had some quiet and good-as-new power supplies and was back up. Since 2001, I take the servers down at least every 100 days or more often during Spring and Summer to dust off the systems, replace fans that are noisy, etc. I still have a spare set of capacitors out there for one more repair.

It's amazing how much dust can accumulate inside a server box when you run it at home. With humans shedding skin flakes, pollen and dust from outside in the garden, those servers were actually like air filters for the house. After cleaning, I was shocked at how warm the side of the case was; and then it hit me that the dust build up had been preventing heat from radiating out the sides of the case too, and that sent more heat out the back through the power supply. So almost every 2 to 3 months, the servers get taken down on a Saturday evening, at 1 am or so, and the cases are cracked open and the units go outside to the patio where I try to stay barefoot and grounded and then blow on the boards with some type of compressed air. Some stubborn soot gets caught between pins or in nooks and crannies like on the cpu heat sink and I use a soft tooth brush with Swiffer dust cloths to clean and wipe. The units go back inside and get tested to see if cpu, case and P/S fans are noisy or wobbling after service. I stock 50mm, 60mm and 80mm fans in 10mm and 15mm thickness (and other size fans too) for this purpose. They get swapped if noisy and I usually switch pin order to bias the fans at 7V versus 12V so as to reduce noise. Not all fans and motherboards support this, so you need to make sure the ones you get aren't smoking or failing to start doing this. Then the systems get closed up and put back into service.

It'll be sad retiring these old friends, since they handle email, web, Java servlets and JSPs, firewalling and database for close to 10 domains and do it without any real performance issues so far. But each consumes about 44 Watts in power when I have other servers that are 1.5 times faster at 19 Watts, which could tremendously improve battery life on the two 1500VA UPS units I own and allow me to consolidate all the switches and routers onto a shared UPS rather than adding separate smaller UPS units for those. With newer, even faster chips using about the same or less power, I may even be able to use just one box for all services and consolidate completely and really saving money, power, the environment, and all that good stuff.

Installation Dilemma - Slim Drive or No Drive?

Ever since Sun produced the Netra X1 line of products, I've loved the low 1U, shallow 13 inch depth case. Those boxes sort of epitomize the whole idea what a small, cool looking, but industrial server should look like. I wished someone made an affordable case in a similar form factor that was a bit quieter, and equally attractive. About the closest thing I can find today is a SuperMicro SC513 or SC512 1U chassis. But at close to $180 for the case, it's not cheap. And to add cost, it requires expensive slim optical and floppy drives. It's certainly too rich for my tastes and I've never investigated the acoustics. Those old Netra's though were pretty quiet, if I recall.

A compromise of sorts is to go with a BookPC form factor chassis. The first gen of these cases were like the ones I described above; they came with small proprietary boards and power supplies and actually were the size of a large telephone book. The whole barebones kits could be gotten for under $100 + shipping. A good feature was they did take standard-sized optical and floppy drives, usually over the motherboard, but it got cramped inside pretty fast. Back in the late 1990's, not all DIMMs were low-profile. Some were 1.4 inches tall and too tall to allow sufficient clearance between drive bracket above and cabling that went over the DIMMs situated underneath or partially below the drive bays. Subsequent revisions of BookPC cases have gotten longer and deeper so the board is entirely clear of the drive bays. Prices have also gotten longer and deeper and they start around $60 and go up to $100 or $200 for some sleek all-aluminum cases. (If I'm gonna pay $200 for an all-aluminum case, it better protect the board from EMP from the next nuclear detonation in my neighbourhood! LOL!)

And the peeve I've had with cheaper BookPC cases are they're really noisy due to all the cheap small fans because the makers know that there will be folks out there who think they can save a few bucks and get a small case for that Quad Core/Quad GPU gaming system. So, yes, that's why they have a bunch of fans and they're noisy. And that's why I focus on finding low-power processors and motherboards to reduce volumetric heat generation on these small boxes. (Note: Before video games, kid brothers would watch their 2nd grade sisters use a Hasbro Easy Bake Oven with the 100 Watt light bulb heat source - it bakes -real- cookies. Understanding the heat generated by a little box might be something to bring back into 2nd grade education so big boys don't grow up and try to shove a 200Watt heat source into a small case. Note2: We could educate the manufacturers too... only, most aren't socialized in America with Easy Bake Ovens - they're just putting in noisy fans to cover themselves and their distributors against too many RMAs on melted chasses).

For all my complaining, the compact BookPC size and shape does appeal to my sense of aesthetics, and with some re-wiring and soldering skills, I can usually lower the voltage on case and power supply fans to make them slower and quieter, yet still sufficent to cool a lower-power system. And that's what I did to a couple Enlight 7396AM1 low-pro cases. These have a high quality, sound insulated chassis, front USB, comes with a fairly quiet power supply and Directron.com has them on clearance for $19.99 with $13.99 shipping, which gets cheaper if you buy more than one case. (See figure below - courtesy of Directron.COM):

Fig. 2. Enlight 7396AM1 micro-ATX case for $19.99 + S/H at Directron.COM

I bought two of these last month, and then realized that the drive bays take a normal floppy, but -slim- CD/DVD drive. I was sort of in a dilemma. I didn't want to shell out $75 each for 2 slim DVD burners when I had perfectly good regular NEC 3550As in stock. It would sort of defeat the whole idea of buying these cases for $19.99 + shipping. Then I asked myself the question if I really needed an optical drive or even floppy on the box, since the reason for these systems were to replace those two 8-year old AMD K6-2 450 MHz bookpc servers. Neither have any optical or floppy drives in them; they were removed after OS installation, leaving just the hard drive.

Chicken or Egg Solaris Install? How about transplants?

So I don't have slim optical drives to do the Solaris install using the small Enlight cases. In addition, the PCChips M863G with SiS chipset boards support Novell IPX Netware Boot but not PXE, so a DHCP boot of the system isn't going to be easy to implement. And even if the board supported PXE boot, the default Solaris install doesn't have the SiS900 network driver. I guess if we had PXE, I could disassemble the x86miniroot in Jumpstart to add Murayama's sfe driver (Howto provided in a previous Blog) and that would give us an active network interface to complete the Jumpstart install. But the boards don't have PXE, only IPX Netware boot so thinking about PXE boot without a PXE boot capable NIC would be moot. I checked my inventory of Intel and 3COM ethernet cards and none had the optional PXE boot ROM. And the only one with the PXE boot option ROM I have in stock is a PCI-e Intel e1000g. Great. More shopping, but that would take a bit of time and I wanted to install the systems, there and then.

So the best solution I initially thought of was to temporarily attach a standard DVD-ROM drive and have it perched outside the propped-open case and do the install once on each disk. That'd expose the system for just an hour or two and we could then close them up. But that's still an hour or 2 or more. Plus the whole hokey setup with cable/ribbon hanging out and a bare drive setting there spinning loudly for 2 hours while I'm trying to watch a remake of Van Helsing with Hugh Jackman, isn't what I had in mind. And looking over my desk, I had a bunch of IDE Seagate Barracuda IV ST380021A drives, in clear plastic clam shells, just sitting there, recently swapped out of a couple of test boxes with Solaris Nevada b52, which got newer SATA drives.

A conversation with a colleague earlier in the Spring of this year came to mind. We were all sitting outside the Sun Santa Clara Auditorium after a Silicon Valley Open Solaris Users Group (SVOSUG) meeting and a bunch of folks were going over to Denny's or IHOP for a late bite. Dan Price had just given an S10 Next Gen Overview and he covered a lot of things. And our SATA team had given a talk on the new SATA framework. And one of my colleagues who works on x86 boot and ACPI was there. I think we were waiting for a couple of guys still inside and probably cleaning up, and I asked my colleague about why if I switch disks from one box to another, I can't get Solaris to boot, except for Safeboot.

My colleague gave me that funny look, like, "Why would anyone wanna do that?" I explained that from a customer support perspective, it'd be cool to flash a Solaris image to disk, then ship it and let it boot up and do self-config. He countered that, no, this isn't a big feature demand for Solaris and asked rhetorically how many folks would ever use that feature anyway? Well, I wasn't sure. But on Linux, I do this all the time - preflash a disk and when I need to do a quick build, I slap the drive in there, Kudzu kicks in, and voila... system is configured in a couple of minutes. Solaris gets stuck in reboot-hell if we try this. So after some debate, which went nowhere, I brought up the Jerry Seinfeld episode that got me to start watching that sitcom. And this was the first episode I ever chanced upon where Seinfeld has his little comedy clips at the open and close of each episode. He's standing there talking about the black box.

"Ya know, when a plane crashes... the only thing that survives is the BLACK BOX... Ever wonder why they don't just make the WHOLE plane.... out of THE black box???" (laughter).

In all seriousness, we know that Solaris safeboot has the hooks to rebuild the boot-archive and device trees, and the installer figures out the devices and puts a permanent map of that onto the filesystem somewhere. So why can't we put those same hooks into the regular Solaris boot or multi-boot? That's what I asked my colleague.

He gave me that, 'James, you're naive and uninformed about Solaris x86 boot'-look and just told me that it's not a widely used feature, and that's not what multi-boot does, and started down the path of how the GRUB works with multi-boot and the whole secondary boot blah, blah, blah, blah. Which I took as obfuscating the issue with detailed specs that don't matter to end users. Anyways, to make a long story short, I was miffed and decided against Denny's, and went home to eat cold supper made by my wife, who's always been a lot warmer and at least seems more understanding.

Fast forward 9 or 10 months and here, I have two perfectly good disks, almost new, and pre-installed. I searched the web and managed to find a few Sun FAQ/Developer discussion board topics. It was clear that at least a couple other folks wanted to do the exact thing and actually had the gumption to ask us how. One guy got pretty far in the boot but still didn't get it going. Our standard answer was to boot the kernel in debug mode and look at the output. That's fine if you're a Sun Engineer and know how to look at the messages and hex scrolling off the console. But to for most folks, it's nonsense and just shows a lack of empathy on our part.

But within the various tips, I put together the various tips and gave it a try; magically, 5 minutes later, it was all working on the new SiS chipset motherboard. Here are my steps:

  1. Boot into Solaris Safeboot mode. You can get access at the Grub menu, usually the 2nd option.
  2. Mount the found Solaris partition on /a Safeboot will usually find the slice on the disk with Solaris and ask if you want it to mount on /a. Select Yes.
  3. Move /a/dev, /a/devices, and /a/etc/path_to_inst to another name (I just append .orig) and then create new directories, (mkdir) /a/dev and /a/devices, and touch /a/etc/path_to_inst.
  4. Run "devfsadm -r /a" to rebuild the device tree
  5. Edit /a/boot/solaris/bootenv.rc and modify the line with "setprop bootpath '/pci@0,0....' to match the path you'll find mounted for /a (i.e. run a 'df -k' command, and you should see /a mounted from /dev/dsk/c1d0s0 or something, then run 'ls -l /dev/dsk/c1d0s0' or whatever your device listed was, and you should see the actual link point to ../../devices/pci@0,0/...) The path to bootpath you want should be the hard disk which is mounted as /a and you just need to find the expanded /devices/pci@0,0/... path and put that in the bootenv.rc file on the Solaris root filesystem on the hard disk (sans the /devices/ prefix of course).
  6. Now run "bootadm update-archive -v -R /a" to rebuild the boot-archive on /a
  7. run a 'touch /a/reconfigure'
  8. Run "cd /; sync; sync; sync; umount /a"
  9. and finally reboot.

There may still be issues if you have changed the IDE priority (master/slave) of the hard drive or moved it to a different SATA socket. In these cases, you probably will successfully boot but run into problems with mounting filesystems not found, in which case, boot into safeboot again, and edit /a/etc/vfstab to correct.

Once the system boots, it does retain its legacy settings for network and naming service which may be totally bogus, in which case, I check for any missing drivers (like an SiS900 Fast Ethernet module) and either transfer the source/binaries for the modules via CD/DVD media or USB which usually will work. I'll then delete any /etc/hostname.[NIC#] files and run a sys-unconfig on the system, and reboot again.

Yes, not the most elegant of solutions, but it could be scripted and one of my colleagues down the hallway thought we could do our customers right but putting that script into the safeboot and documenting that we put it there. For now, I hope the instructions help others who might be in the same boat. It takes about 5 - 10 minutes to swap the drive and reboot and reconfigure the system if you know what you're doing. That's a lot better than an hour or two re-installing or upgrading and it also means I can keep pre-flashed drives just sitting around to save time on installs and testing.

BTW, Solaris b52 runs well on the PCChips M863G motherboard with AMD Geode NX 1750. The SiS 741GX/964L chipset functions normally, and Xorg even finds and configures the SiS onboard graphics with no work required. The graphics support on the SiS chipset, while supported in Solaris, is really grainy. And the graininess is particularly worse on one of the systems than on the other. I guess I could stick an AGP card into in the slot (and I did try an older ATI Radeon unit and it looked beautiful), but it eats more wattage with the optional graphics card. And since my plan is to mainly run these headless, I really shouldn't care. But this isn't the first time I've had graininess issues with Integrated graphics, especially on these small form factor boards. I suspect it has something to do with the number of layers in the board (usually cheap boards use fewer layers) and the integrity of the analog/vga signal that goes through the motherboard where there may be a lot of RF interference to get to the back I/O plate. I've used the same graphics chip and software driver on different boards and sometimes the image is crystal clear. And of course, with an optional AGP card in the slot, the board is raised and separate from the board and less likely to have interference due to wires too close in proximity. I get similar graininess with the VIA mini-ITX systems, but it's not as bad as on the SIS chipset on this particular board, plus an older celeron system I used to have with an SiS chipset and SiS Mirage graphics looked great, so it isn't the chipset.

The goal is to run them as network servers, so I plan to disable the graphical login and run only in text mode anyway. Next, I compiled and installed the sfe-2.2.0 gldv3-nemo driver from Masayuki Murayama's Free Solaris NIC collection and it just works. Honto ni, arigatou gozaimasu, Murayama-sama! (someone send this guy a case of Sapporo Nama Biru and Pizza Hut vouchers or a free Shinkansen Ticket to Sapporo where he can pig out at the beer factory at the "Ghengis Khan" Mongolian BBQ Tabe/Nomihoudai [all-you-can-eat and drink] !)

Note about Xorg on older Unichrome (not Pro) Graphics

I mentioned above I had an issue with graphics installing Solaris on the VIA Unichrome graphics. This was on the Biostar M7VIG 400 board. Back in Nevada b30-something timeframe, more than a year ago, I was having some graphics issues on my EPIA mini-ITX boxes which only impacted the graphical installer, and the VGA text modes. But Xorg, itself, worked fine. Only, if you selected Text Console as the login option, you'd get a dark and illegible screen or sometimes a pastel screen. But the version of Xorg did work and very stably if you always used the graphics. We had a few talks with some of the VIA folks who tried to reproduce this on newer mini-ITX boards and they didn't see the same problem. The funny thing was we knew this was partly a problem with the Solaris driver and partly in hardware rev, because in going from an older rev of a PCChips M789CG v2 to a newer rev, v3, both with VIA Unichrome graphics, suddenly the problems went away in the same build of Solaris and all the modes: the VGA text, console login and X graphics worked fine - no driver changes. A friend mentioned also that a 1.3 MHz Nehemiah on CN400 chipset with Unichrome worked fine, but I saw weird pastels or garbelled text consoles on my older EPIA ME6000, M10000, and EPIA 800 systems.

Some time around the build 48 time frame, I was going through and upgrading a bunch of my systems when suddenly, all of my VIA Unichrome systems would power-up in blanked out graphics mode (i.e. there was no VGA signal coming from the graphics port and the monitor would blank with a yellow blinking standby-mode light). The behaviour was very peculiar. And nothing was showing up in the logs. I tried to log in, and yes, it was as if X was thinking that it was running because I could see the disk spinning as if I succeeded in logging in, and ssh'ing from another machine showed that I had active Shell processes on the console, even though the console was blanked (like someone turned off graphics or blanked the screen in low-power mode or something like that).

Back in the old XFree86-to-Xorg transition days of Linux, I used to run a bunch of Biostar M6VLR boxes with the old Trident Cyberblade graphics embedded as part of the VIA PLE133T chipset. Fedora Core 1 and 2 had fits with the Trident Cyberblade. And even FC3 still had some issues. A more stable driver binary was available and the trick was to use it to replace the default driver module used by the Xserver. Using the same trick, I loaded a Solaris DVD from a previous build and found the ./Solaris_11/Product/SUNWxorg-graphics-ddx package and copied the ./archive/none.bz2archive to /tmp and unpacked it using:

        # bzcat none.bz2 | cpio -C 512 -idukm

This created /tmp/X11 and inside /tmp/X11/lib/modules/drivers/ was the via_drv.so file.

I copied the old VIA graphics driver in /usr/X11/lib/modules/drivers to via_drv.so.orig and then copied the one unpacked in /tmp/X11/lib over to /usr/X11/lib, clobbering the old version. Because the sizes differ slightly with each build of Solaris, I couldn't tell which was the same or different. So I methodically went back checking each build of Solaris until about build 42 or 43 before I found a version of via_drv.so that worked with the graphics and didn't blank the screen. But it did cause the Biostar board with Geode NX cpu to revert back to bad console text. So at least it seems that the Biostar boards were old versions of the VIA Unichrome hardware (reaffirming my disappointment with that first Geode NX/Biostar combo). But I had a work around to get X graphics working.

I know build 53 put back some big changes to graphics, and improves AGPgart. I don't know if it fixes other graphics issues; but it's worth a try. And b54 just got posted internally. Unfortunately, that Geode NX box is up near Vancouver, Canada right now and in a powered-off state with no LOM (lights-out management). I'll be up there for Christmas in a couple of weeks and then I'll give it a try, unless the weather warms up a bit to let me get out there and go fishing for some winter steelhead. I'm usually more worry free in winter while fishing. The black bears are hibernating and not likely to be up stalking me while fishing. I'll save that for another blog.

Monday Dec 04, 2006

Solaris x86 Desktop Hardware Shopping

I'm not sure if it's the Christmas holiday coming up soon, or if it's just coincidence, but the 3rd person this week emailed me this week asking for recommendations on a Solaris x86 box. Two were asking about low cost laptops for kids that will dual boot, and the other was for a home server. There's also been this interesting internal discussion on small form factor Solaris x86 boxes that's garnered a lot of enthusiasm and suggestions for small appliance systems.

I'm by no means the expert on hardware, especially the high end. I know you can buy that premium stuff online at the Sun shop which has a new look and feel. Surprisingly, I needed to do a double take when I just looked at this site, just now, because while I sometimes have a disdain for big-iron workstations, a dual core Opteron workstation starting at $995 from Sun has to be a printing error, right? It's missing an extra digit somewhere. No, it actually looks like a real price, and it looks like it comes with 1 yr hardware support too. That's pretty hard to believe coming from Sun. We're actually selling them cheap. I'll need to look further into the shop to see what other stuff we sell. But back to the really low-end do-it-urself boxes and shopping for Hardware.

I've been known to have a weakness for big sales (except ones requiring rebates), and I have a hoarding instinct for this computer stuff. Some folks refer to this as a disease. And yes, admittedly, I've got some variation of the illness. But I reassure myself that I'm not the worst. I do have lots of spare parts, but it's all well organized into bins and boxed and bagged properly in anti-static bags. And I tend not to carry anything expensive or power-hungry. For example, the fastest CPU I keep in stock is maybe an Athlon-64 3000+, and I only have one of these. And the biggest graphics card is maybe a 128MB AGP8x unit. The rest is slower, really cheap, but extremely usable and great for fixing computers and great for building Solaris boxes. I'm referring here to my home inventory; not my work, where we get to play sometimes with some bigger, enterprise type systems.

And my stockpile of stuff has come in handy on occasion, like when my nephew's budget "educational software" (i.e. gaming) system died recently due to a bad power supply that then led to a fried motherboard. I had both a compatible micro-ATX motherboard in stock, plus a newer, quiet power supply too and he was back up and running; Or when my Dad's box suffered back-to-back power outages during a recent late summer rain storm that shorted out the power supply, I had a surge protector and new special TFX12V power supply in stock, and still have another in stock just in case. Or if my psychic powers are right, I predict my Bro-in-law will have paging issues with a particular PC game that is memory hog and that extra stick of 512MB DDR333 will be the salvation he will be looking for... just mark my words. I've got it in stock and people will need it!

If you stockpile more than 5 boxed motherboards, or 5 new cases, or more than 5 separate power-supplies, then probably you fall into the class of electronics junky known as a "Hardware 'Ho". The etymology, I'm told, is not from anything related to the "Ho, Ho, Ho..." from a cheerful Santa at Christmas, but from the slang term for "Whore" implying, you'll pay anything to get close to computer hardware. I suffer similar issues with being a Fishing Tackle 'Ho, and I'm also a recovering Bicycle Gear 'Ho. But that said, there are ways to live with being a HW Ho and not break the bank. For example, one way is to focus your energies on low-cost, low-power stuff that works well with Solaris x86. That's how I channel the urge spend all available money on hardware on the biggest, baddest and most power-hungry systems into a contest of who can achieve the best price/power/performance. So, for example, instead of say, blowing $650 on a graphics card to get better frame rates on Quake 4, I simply evaluate 3 integrated graphics motherboards under $50 and experiment with which one has the best graphics performance for the dollar. Another part of my therapy is to go to Fry's for lunch (their small cafe inside the store has surprisingly decent sandwiches) and buy no hardware. The goal is to resist the urge to spend money frivolously while being tempted by a store full of stuff ready to buy. Becoming a discriminating shopper for Solaris-compatible stuff is a good way to curb that urge to waste money, and we'll go more in-depth below.

Step 1: Solaris End-User Requirements

As far as I know, Solaris isn't really an OS for gaming, at least not yet. It's possible to dual boot a Solaris box with Windows so that you can have an awesome PC gaming system, but get real work and do real Solaris development using the same box. But as a home-user, I use it for word processing, spread sheets, some digital photo stuff, email client, web browser client, playing and ripping audio, as a file/web/email server and firewall. I also VPN into work from home using IPSec and I burn a lot of CD and DVD iso images. I find these were most of the things I used to do with Linux, but less and less because on Solaris, the software now performs as well if not better than Linux, and has pretty solid stability even across kernel versions. I can tell that it won't be long before, at least in our household, Linux won't be active, although, I plan to always keep a current copy of a popular free distro around to play with.

More than just software and performance requirements, I also have environmental requirements. My living room and bedroom are places where these systems will stay. People coexist here and must endure the noise. People living in a place also stirs up air and dust too, and the PCs need to behave stably with lots of dust fouling during hot summers and cold winters.

If your requirements are like mine, nothing there really needs a lot of big-honkin' iron processing. Some of the audio ripping and conversion, or editing photo images does take some CPU and memory, but for the most part, any Pentium-3 class machine with more than 600MHz and 512MB of PC133 SDRAM is almost adequate for Solaris. Anything much faster, will be spent idling, or burning more Watts which costs money if folks actually stopped to think of the costs to power and cool a system. In addtion, it could shorten the life of the system because that extra power requirement goes into more fans to suck more air in to cool the system, depositing more dust that can foul and burn out power supplies and overheat capacitors on the motherboard causing failure. Some of my colleagues refer to their home racks as "air filters" for the home, since they suck up so much air and deposit soot inside the case.

This doesn't mean we should just stop buying newer, high speed hardware. In fact, quite the opposite. While the CPU and memory might have achieved performance adequacy 5 years ago for home users, some newer systems may provide even better power economy while running faster, as well as support newer, faster chipsets, graphics and monitors, faster networking and USB peripherals, and are likely to be more available in stores at a lower price, than having to try to find it on some auction site because the original part is no longer made. Sellers rarely make overt offers for low-power, high-value hardware; you need to look for these deals.

Step 2: What works with Solaris x86

To find out the official list of hardware items that work with Solaris, there's an official site called the BigAdmin HCL. On that page, you'll get a list of servers, desktops, motherboards and laptops that are certified or reported to work with official Solaris 10 and Solaris Express. The HCL allows companies and individuals to buy a Support Contract from Sun for an annual fee. Using the OS is still free. Folks can also download the Hardware Compatibility Test Suite (HCTS) from Sun and run this on their hardware and submit their entries. Version 3.0 of HCTS recently went public and it runs for about 12 hours and tests the filesystems, cpu, memory and network pretty intensely during that time. It requires two machines (a system-under-test SUT, and a Test Manager/Server) hooked up back-to-back, or through a seperate switch on a private network to run the fully networked test. The Solaris Marketing folks used to have a Tshirt Give-away for anyone who registered, downloaded the HCTS and tried to run it, then made a Report-To-Work submission or certification. I made a bunch of submissions, but found out later than Sun employees were forbidden to participate in that contest, and anyway, they didn't have 3XLT tshirts to fit me. Bummer.

Unofficially, Solaris x86, especially, the latest build of Open Solaris, supports a lot more hardware. In general, Solaris will run on most major CPUs (AMD, Intel, VIA) and motherboard bridges and north/southbridge chipsets that support Intel's x86 instruction set or the AMD64 instructions (or variant, such as Intel's version - EM64T). This includes standard I/O like IDE/ATAPI interfaces, any SATA interfaces that support IDE legacy mode, systems with PCI, PCI-X, and PCIe bus, older ISA bus support for PS/2 keyboards, mouse, peripherals and Serial and Parallel I/O, most on-board USB, and systems with an AGP slot. Typically, Solaris can be install in console mode on any x86 system with about 128MB of memory, some type of ATAPI optical drive and some type of supported hard disk with at least 5GB of free space. But if you want something usable, 512MB or more RAM is good, and 80 GB disk might be a start. By default Solaris x86 auto-senses 32-bit or 64-bit architecture and boots the OS as appropriate. And yes, the 64-bit version of OS is fully compatible with 32-bit user applications.

Where Solaris doesn't work out-of-the-box is for certain on-board and common peripheral devices like new native SATA and RAID controllers, Wireless 802.11, some ethernet chips, some new graphics cards, or and with some types of high-definition (HD) audio devices. Solaris also lacks software and drivers with proprietary peripherals, like Point-of-Sale scanners, printers, card-swipes, kiosks, etc. Many network, audio, and storage controller drivers are available from third parties, and in the open source space and there are companies that sell proprietary drivers solutions that run on Solaris as well. There are a few drivers in the GNU GPL space that Solaris might be better off having, but those are getting fewer as hardware vendors are coming over to Solaris and we apply more engineering resources for porting. For the time being, it still pays to stick with what we know.

Step 2a: Motherboard Selection

With some experience, we find that most chipsets work with Solaris and it isn't really a particular maker or brand that causes incompatibilities, but it's an onboard peripheral bundled with that particular chipset that may not have a supported driver. When that is a native-only storage controller or NIC or Graphics port, then immediately the Solaris install is more difficult. So the trick isn't so much choosing the maker or the chipset, but knowing what peripherals come with a particular board or chipset-bundle. In some cases, the behaviour of a peripheral, like a native SATA controller, can be adjusted to set to use legacy-mode only, in which case, the board works fine with Solaris.

I like the low-end and all-in-one chipsets because this category provides excellent price/performance and low-cost to buy and own. Counter to what others tell me, I actually think the cheapest and most mass market motherboards are some of the most abused and tested board on the market, and therefore, manufacturers tend to make them more robust to human abuse than on server boards that cost 10 times as much, but have a much smaller marketshare. Certainly, the margins are much greater on server hardware (yeah, we at Sun should talk... we're in that business). But the whole ideal of achieving volume Solaris for the Proletariat means (a) making it cheap to get the OS, and (b) having it install and run well on a damn cheap box!

Rather than go through each chipset and what works, it's probably easier to say what is not likely to work. Since most CPUs, bridges, buses, and standard I/O controllers to memory and disk have to be somewhat compatible with Windows/x86 instructions and memory flow, for the most part, Solaris just works and recognizes and uses that hardware like any other x86 operating system. Standard drivers are in place for the usual ISA legacy devices like serial, parallel, ps/2 mouse and keyboard. Also supported for the most part and standard USB devices and USB storage, and ATAPI drives like IDE disks and CD/DVD ROM and Burner drives and any SATA drives that are running in Legacy IDE mode.

Step 2b: Graphics Selection

What might NOT be fully supported in the install media are the graphics, NICs, audio devices, and WiFi networking chips. These don't necessarily mean that all is hopeless. Some drivers can be downloaded from the web if another system is available and networked and you have something, like a USB jump drive to copy drivers after the initial install for configuration. In other cases, there may be no support for the onboard device, but you might be able to install an add-on card that works with the board you have. Graphics is a good example. Up until build 53 of Solaris Nevada, the AGPgart driver didn't support the onboard VIA Unichrome Pro graphics on any of my newer VIA systems. But those systems all had optional AGP4x/8x slot or PCIe and it was possible to buy a very inexpensive ($15 - $20 online) ATI- or nVidia-based AGP card from a previous chipset line (e.g. ATI Radeon 7000 or nVidia MX-4000). Laptops are different story when it comes to graphics - pretty much, you're stuck with what you get. In these cases, Solaris supports the slightly higher end embedded graphics chips like the ATI Radeon Xpress 200M or the nVidia GeForce 6100/6150. Solaris has lots of issues with Intel embedded graphics, even among chipset variants and vendor implementations that should be supported by the Xorg Intel embedded graphics drivers. I often have the same graphics non-compliance issues with Linux, and luckily, it's been getting less frequent with both Linux and Solaris, as it should since both leverage the Xorg source. But I try to avoid embedded Intel graphics on a system for running Solaris graphics mode. But ATI and nVidia seem like well support brands these days. We even support the newer nVidia Quadro line in Solaris.

Step 2c: NIC selection

With NICs, the problem is less difficult. There are many free drivers out there for Solaris. Murayama-san puts out a bunch of open source, free Solaris NIC drivers. So the installation may not find any drivers for the ethernet devices onboard, but if you have a VIA, SiS, Tulip, Davicom, or other mass market embedded NIC, then most likely, there's a driver for it. What folks may want to avoid are the newer embedded PCI-Express GigE NICs. The drivers may actually use the same core logic, but the PCIe bus signals and handles interrupts differently over PCI. We do support most high-end PCIe server NICs, but each has a specific initialization and so this is device specific and our driver requires some amount of code to correctly initialize each variant of that MAC. So it isn't as easy as trying to stick that vendor/device ID into the /etc/driver_aliases file and run devfsadm to initialize one of these PCIe embedded NICs. It may work, but usually not. Luckily, if you need a NIC and this isn't some laptop where you have no choice, then you can get a Realtek 8139-based Fast 10/100BT PCI card for about $0.99 on sale (sometimes $0.49), and I've purchased 3 Realtek 8169 10/100/1000 GigE PCI cards which I paid $4.99 for each. This is brand new. Not used.

Step 2d: Audio Selection

Solaris 10 and later supports most Intel Spec AC'97 audio, VIA 823x Audio, and some legacy Sound Blaster stuff. That covers about half of all the audio on most boards these days. There are still a few boards with AC'97 audio which the Solaris included install media won't recognize, and there is the new class of High Definition or HD Audio chips. For folks with an unknown AC'97 type of audio controller/codec combination (read your spec sheet or prtconf -pv and find the PCI vendor and device ID and web search that device to find out what it supports), most likely, you have a good chance if you download Jurgen Keil's audio drivers and install them. Jurgen supplies his own versions of VIA and AC97 Intel audio drivers. These have different module names and will not clobber the existing drivers, and his installation package does a good job of pruning and transferring PCI vendor and device IDs to his driver control where both the Sun drivers collide with his drivers. Jurgen's drivers are especially useful with some AC97 codecs that have a non-standard sample rate conversion hardware. Where the Sun audio810 driver fails, or takes a huge performance hit going with a software sample rate converter (am_src module), Jurgen's driver allows one to compute the proper sample rate and set it as a configurable in the /platform/i86pc/kernel/drv/audioi810.conf which usually gets the driver to play audio at the proper speed. The information on how to do this is actually in the .conf file itself and has come in handy on a Compaq motherboard with Intel 815 chipset and AC97 audio. Jurgen's VIA drivers eat a bit more memory for buffer space, but lower the interrupt rate dramatically, eliminating almost any hiccups on some very slow first-gen EPIA 500 fanless C3 mini-ITX systems and allow those to play flawless audio.

As of Nevada build 44, Minskey Guo and the Beijing team did a put back of a quick High Definition compatible audiohd driver implementation. Minskey says he was "helping" me, but actually, he did most of the heavy lifting on the all-nighter coding and testing. I ought to send this guy a case of quality beer and certificates for Pizza Hut in Beijing. I wonder if they deliver like in Tokyo?

HD audio comes from a 2004 Intel spec and is pretty much different and not meant to be compatible with AC97. First of all, the spec which is available from Intel's site separates the combined AC97 controller/codec into separate HD audio controller interface and compliant codec. Capabilities are for much higher bit rate (upto 192kbps for 8 channel - 7.1 audio) versus the old AC97 (48kbps for 6 channel - 5.1 audio). This allows motherboard vendors more performance and choice in selecting the codec and implementing rich audio support for multiple devices, playing different streams simultaneously. At the same time, if the driver is implemented with good support for codec discovery, it's possible to implement a universal audio driver that can ideally parse and initialize any codec. All communications between audio applications to the device are through the controller and through defined kernel DMA structures. This eliminates applications ever needing to hold onto pesky base address register values once the memory segments, driver and codec initialization are complete. The only problem is that support for codec discovery isn't trivial and can take time to test and properly debug. Also, we learn in the hardware business that every codec isn't quite the same and there's always some errata on either the controller or the analog pinouts the codec is hooked up to, such that the discovery may not provide correct information. It's quite a bit easier and faster if the register specs are available to simply check for codec device and do the initialization in the driver code or to use an external configuration file that enumerates name=value pairs of properties that specify where the standard pins for that codec are. It makes it easy for vendors to ship audio codec initialization files too as properties. But that all depends on if vendors are willing to give out the specs on their codecs. Which is really the crux of the problem, since that's been somewhat hard to get except for the Realtek, which have just been excellent Solaris partners. So far, the Nevada audiohd driver can support most Realtek HD audio codes like the 260, 880, 883, and 885 and others. And this supports stereo audio playback and recording. But the number of tested HD Audio controllers isn't great. But if you have a Realtek codec or think you may have one, you may still be able to get the driver to work by editing the /etc/driver_aliases file, and searching for "audiohd". Then add an entry in the same format for a new device. For example, on my MSI K8NGM2 motherboard system with MCP51 chipset and HD Audio controller, the PCI device node blocks me from seeing the codec directly, but a "prtconf -pv" shows the controller as "pci10de,26c" which I added to the list, then ran devfsadm -i audiohd, reboot and suddenly, the audiohd driver works great. The driver may work for quite a few other audiohd controllers that are back-ended using Realtek HD Audio codecs. But we didn't have too many systems at this point to play around with, although a number of laptops are now coming out with HD Audio standard. Many don't use a Realtek HD audio codec, but some other brand, like a SigmaTel.

Support is unlikely until we get specs, or if we have time to work out a really good parsing algorithm for codec discovery. But for the time being, you're not all shot on Solaris if you have HD Audio. In fact, there may be a version of OSS (Open Sound System) from 4Front Technologies that works on Solaris x86. 4Front has been doing high quality commercial audio drivers for UNIX for years. And their latest stuff is free for personal use and has a 6 month license that is renewable for free every 6 months, or can be purchased perpetually for just $49 to end the hassle of re-submitting for that license key. I've tried the OSS stuff and it does work with the ATI SB450 HD Audio controller and codec underneath on my new bargain Toshiba M115-S1064 laptop. The audio worked fine out of the box, but had some issues with standard Solaris audio support devices (like the Gnome volume control) not responding on the laptop (since there is a manual dial for audio volume), but otherwise it sounded good. I have some poorly ripped Enrique Iglesias and Led Zepplin tracks that had lots of Cymbals and high pitch audio that got a bit over saturated and tinny in this driver. But I peeked at the hdaudio.conf driver (no collisions in the name space again) and I downgraded the quality of the input audio stream the driver was expecting to medium (instead of the default - high) and suddenly, the tinniness went away and I was hearing great audio again. It pays to try different settings in the driver if it has a .conf file. Hopefully, between the existing drivers, Jurgen's drivers, and OSS drivers, most audio chips will just work with Solaris.

Step 2e: WiFi Selection

I used to think selecting a WiFi chip was specific to laptops. But recently, in Sunnyvale, the Mtn.View based company: MetroFi Wireless, has installed a bunch of WiFi base stations. These are advertising supported services, but it's changed my outlook on how to help the families of kids at my local elementary get online using broadband. Instead of deploying a donated and refurbished Linux Box with some semi-expensive and rare controller-based modem for a cheap dialup service, the PTA might be able to collect donations and get some cheap PCI WiFi cards, install them into these systems, and then use wifi to get on the network with no monthly or annual dialup fees. We wouldn't be limited to Linux and its LinModem support, which is mercurial at best with the plethora of Host-signal processing (i.e. controller-less) modems. We could go stock 802.11g and with the right booster antennas, users should could get good WiFi signal.

That's motivated me to look more closely at WiFi support and not just on laptops, but at built-in, cardbus, and PCI card WiFi NICs. A big list of WiFi drivers recently got into build 53 of Nevada. These support PCI, mini-PCI and cardbus versions of popular 802.11b/g chipsets including the Intel Pro Wireless 2100 and 2200bg/2915abg, the Prism/Orinoco chipsets, the Atheros 52xx series, the Ralink Technologies RT2500 series chipsets, the Realtek 802.11b wireless chipset and the Cisco Airnet chipsets. The Atheros driver works with most of the previous mini-PCI laptop ABG cards. It doesn't quite work yet with the newer AR5006 PCI express line which is starting to become more prominent with laptops. The same story goes for the Intel ipw3945 which is also a PCIe device showing up in lots of laptops, and latestly, the Broadcom BCM4306 line of WiFi chips fills in the remainder. So none of the newer Wifi chips are supported... yet. But my bet is on the AR5006 since it's already available in the MADwifi driver on Linux so it shouldn't be too long. I don't have too much public news about the ipw3945 or the bcm4306 except that we don't have the specs to write the driver, but some reports claim that the NDIS wrapper can provide a usable and stable driver for either device on most laptops.

But besides laptops, what's interesting is the availability of WPA on top of a lot of PCI-bus WiFi-BG cards. To my surprise, a lot of the cheapest (e.g. Airlink 101) PCI WiFi cards are based either on the Atheros (Super g) 5212 chip, or the RaLink RT2500 chipset (regular G). These are cards that show up frequently on sale for under $20 at stores like Fry's. And the RT2500 version have been down as low as $7 for the card (limit 2 per customer) which is why I have two at home. For folks that don't want to rewire their homes with Cat 5e/6 cabling, and have noise already on the PowerLine and can't get more than 6Mbps/sec throughput on PowerLine accessories, then 54 Mbps is a refreshing increase and it only takes switching to wireless. So Solaris users can simply go out and get a cheap card for their desktops too and enable WiFi without shelling out a lot of money to go wireless. Not bad at all, I'd say.

Step 3: Where to buy this stuff?

I shop for most of my stuff at these sites:

  1. www.newegg.com - probably where I spend the most money. Awesome website, great customer product reviews, and awesome prices. Once you New Egg...
  2. www.ewiz.com - less selection than newegg, 5% cheaper on boards than newegg in a lot of cases and a local Bay Area company with insanely fast service - order it and get it in 24 hours!
  3. San Jose Mercury News - click on the [Newspaper Ads Online] link and see daily Fry's ads. If I see something cheap, a quick trip at lunch with 3 other colleagues in a carpool is the usual menu/venue. And you can order a decent lunch at good prices at the Fry's coffee shop inside the store.
  4. www.directron.com Another low cost online retailer. Houston, Texas-based. No sales tax to CA and good service. I get a lot of quiet-PC gear, adapter, fans, and cases from them.
  5. Starmicro.Net A Bay Area company with fast service and some really cheap prices on older memory and cpu.
  6. www.compuvest.com A Washington-state based company with a long history of low prices and good mail order service. They carry a large variety of older technology and new stuff at competitive prices and out-of-state means no sales tax either.

This is by no means an exhaustive list. Sometimes TigerDirect.COM has a good deal if you're into waiting for Rebates (I'm not), or SurplusComputers.COM will have some good deals too which are online-only and not available if you go to the store for pick-up, which is a bummer, since I live pretty close to those guys. There's also www.axiontech.com, another Texas-based company that caries some good motherboard prices, just to name a few.

I tend not to shop eBay that much. There used to be deals in the past, but most sellers have jacked up the price on shipping and handling to the point where the prices are ridiculous. Often, with computer gear, the shipping and handling are more expensive than the item. Who in modern history would pay $25 per disk for a -USED- IDE 40GB drive? But that's exactly what it comes to with these online auctions and the bidding wars in the last few minutes of every auction close. The going rate for disk storage is about $0.40 per Gig and that's out-the-door prices after tax and shipping on a -NEW- disk drive retail. I usually wait for a sale on some Seagate 5 year warranty SATA or IDE drives and get storage at close to $0.25/Gig. Folks are paying upwards of $0.60/Gig for used stuff and that's just stupid and buying from some kid with a 97.9% rating. For that amount of shipping and handling, I'd expect a 110% rating if it existed. But again, eBay does serve its purpose for those desparate and in search of something the other vendors don't carry any longer. I picked up a couple of Intel D815EEA boards for $16/each with shipping recently to build a couple more Solaris boxes to go with a couple of Intel PIII 733 MHz cpus I bought for $8 each at Compuvest.

My Top Pick for Solaris System Right Now

Clearly, there are a lot of choices that anyone could make right now about what parts to throw into a Solaris box for home use. And it's a lot of fun to build and install systems, especially if in the end, they go to help families get low-cost broadband. But I do have a favourite box for at least I think this is the best deal and the one, right now, I wish I had in my home. It appeals to all my personal criteria for the right blend of a) price, b) performance, c) quietness, d) power consumption, e) Solaris support and f) aesthetics. It starts with the PCChips V21G flex-ATX motherboard with soldered-on VIA c7 processor at 1.5GHz. It has a new CN700 chipset that supports IDE and SATA and has integrated VIA Unichrome Pro graphics, VIA rhine Fast Ethernet, and DDR2 support. The motherboard and cpu all-in-one is just $68 at ewiz.com and installs Build 53 of Nevada without any real issues. I'd stick a 120GB Maxtor SATA drive with 8MB buffer which was on sale at Fry's for $49 recently, and then a 1GB stick of DDR2-533 Kingston memory for $95, and put that inside an Antec Minuet 300 Case w/ 300 Watt P/S for $79 with a silver bezel NEC 3550A or 7170A-0S DVD burner for $30 and a Multi-format, Silver bezel USB 2.0 flash reader for floppy drive bay install for $14. Total cost would run upwards of $380 when all shipping and handling was done. Not the cheapest build possible, but a very usable system that's small and relatively quiet. If I were going for a true budget server system and forego a DVD burner and get a cheap case, I'd probably use an Enlight 7396AM1 BookPC case with 180W P/S and save $50 on the case, $44 on optical drive and flash reader. I'd also get a stick of 512MB DDR2 for half the price and save $50 since most of the time, I would run this without X. That'd save a considerable amount of memory more than ample for a simple httpd/j2ee server and mail server. That'd make a box for about $250 that could serve up a lot of services.

How would you install a box without a working network interface (yet) or an Optical drive?

I'll talk about options for that in the next blog.

Wednesday Nov 29, 2006

Solaris Install Experts - the New Chic

Quite a few years ago, I met up with this big guy at Tokyo University. His name was Ohno-san. He had a similar build like me. Big, round, husky. He road a big Motorcycle that really ate gas - sort of a status symbol as big Honcho for being one of the early contributor's to Japan's WIDE network. I remember his greeting to me - the once over stare and then grin and a modest handshake. Not a limp-fish, and somewhat out of character for a Japanese person. He said immediately that he could tell I was a techie/geek and I must know my UNIX systems, since most of us are pretty big guys with similar builds that look like we exercised our forearms lifting slices of cold pizza for most of our lives. But I lacked the facial hair and bad hair. Oh well for stereotyping folks like us. But I don't mind the status and any implied chic other folks -think- I may possess.

There seems to be a revival of that UNIX chic these days. It's not just inside the company. I see it with some vendor/partners and some of the academic/EDU folks. Solaris is cool again, and folks who can install it and fix other folks computers have a certain chic. Granted, we're a couple levels below the God-hood of a kernel developer, but we interface at a higher level with desparate folks in management and marketing who want to try out Solaris but haven't got a clue how to install it properly on their system.

Pepboys (Geek version): Computers Like Us - Colleagues Love Us

Solaris installation is a lot like car maintenance. Almost anyone could probably pick it up if they had the inclination to research a little bit and try to exercise some Emersonian Self-Reliance. The argument against everybody doing this (within a company or outside even) is the notion of Comparative Advantage. This idea says that everyone has their personal strengths and contributes in their own way. Folks have proven mathematically that Comparative Advantage allows multiple parties to optimize their productivity so all sides can benefit. And hence, we all specialize in our particular fields. And it's just my long-winded explanation of why there's a Chic associated with being able to install Solaris. Simply because folks like us are in demand. And so the ones who are less self-reliant will want to schmooze with us to defrag their laptops and partition a slice to install Solaris on it.

But it's still a lot like car maintenance. And a cornerstone in the mechanics trade has been that customers go back to the mechanics they trust, and with an evolving relationship, customers grow to respect the journeymen with lots of experience. Solaris installation ain't much different than doing tune ups. The more systems we get to work on, the more tips and tricks we learn. One of the things we pick up is what to buy, what works well, and what has good price-performance. Yes, it applies to names like Toyota and Honda too, but we're talking systems, and it isn't always the high-end chipsets that are well supported by Solaris.

In-Flight Across the Chasm

And whether that compatibility is a result of more community users hammering on the platform, or that the platform is more compatible and therefore more people are using it, it's clear that Solaris's recent popularity is coming from the x86 side. Sure, it runs on SPARC and we try to insure that out-of-the-box, Solaris just runs well and tuned on SPARC. But our SPARC customers just expect that and depend on it. The Solaris x86 side has been more mercurial. Instead of specifying specific supported hardware (which we Sell), we've had to provide an OS with broad support for many 3rd party devices. Linux has done a great job crossing this chasm of device driver support. Many vendors are providing drivers up front now for Linux. But the Linux kernel and headers are GNU GPL'd and the license can be somewhat severe for enterprises that have trade secrets. Some vendors have tried to play a risky game using shim layers in their device driver to insulate themselves from the GPL. But if they do it right, they end up with massive build environments for Linux because to support compatibility with the ABI, they need to maintain copies of kernel source, headers, and compilers. If they do it wrong, like a couple of embedded switch companies in Europe recently, then they may be forced to open source all their proprietary software on the device which also leverages GPL code or face a massive recall of all sold network appliances in the last several years.

The problem isn't so bad with GPL applications living outside the kernel. Applications are fairly safe if they only link to high-level libraries. But device drivers are kernel modules and live in the same process space that the kernel lives in and rely on GPL headers to compile. To get around GPL, shim layers of GPL code that then link to standalone proprietary object binary code seems to be the standard these days. But compiling a kernel module such that it has no errors in loading (e.g. the kernel taint statement), doesn't work well. The solution, if done right, is to compile a target object binary driver module for that explicit kernel with that explicit distro of Linux with that specific version of compiler, just as an added safe measure. But this grows into a support nightmare for vendors pretty quick. I support at least a couple of partners who support several popular distros each keep about 29 GB and 40 GB respectively in build environment (yes, Gig as in Billion). All this to prevent the taint message from showing up during modload. So it's not surprising that from where my group sits, more and more vendors are actually trying to contact us about driver porting to Solaris x86. There Solaris driver build environments are just a couple to several megs. Mostly documentation and make stuff. Not actual code.

And so, Solaris is trying to cross that chasm today. Fortunately, I can see the other side, and as Joerg Schilling predicted about 1.5 years ago, he said if we kept working on x86 at this rate, we'd cross that chasm pretty soon. Thanks Joerg for believing in us. We're not there yet and so we're still working really hard. But it won't be long now. Along the way, some of us have picked up some useful ways to get Solaris up and running on our systems.

Installation Tips and Tricks - a Summary

It's not all about the drivers. There some other pre-requisites to installing Solaris that should be covered. I'll be blogging more about the tips and tricks of trying to install Solaris in later blogs. But it starts with hardware choice and what not to spend money on. I'm into frugality, and it's often the cheapest, all-in-one motherboards and hardware that have support these days. It's more about selecting the right chipsets on a motherboard. I'm not a big-time gamer, so I don't go after the super high-end market. Plus, if I'm running Solaris, most likely, I'm doing simple home stuff - like audio, some digital camera stuff, some word processing, running some web sites and mail servers, firewalling the rest of my house network, etc. I make it a point not to spend more than $50 on a motherboard. If it's on sale, even better. And if it's a combo with all-in-one graphics, audio, LAN for under $75, even better.

It seems like the Optical media install is the de facto standard by which we judge usability. But more and more, I use network installs. It's amazing how many motherboards support PXE boot these days, and from a previous BigAdmin article, myself and a colleague tried to put a quick cheat sheet on how to set up a network install server and add more drivers to the netinstall image. But did folks know that they can pre-flash disk drives with Solaris on them and then re-configure them? I have a couple of servers at home and in the office that don't have an optical drive and don't have PXE boot bios extensions. I installed Solaris on a disk stuck in an install machine that flashed a netboot image onto the disk. And then I stuck it into the server box with no optical drive. There are issues of course with the boot-archive, the old device tree, etc. Linux does really well in this area with their Kudzu and admittedly, I wish Solaris were better. But installation isn't something most folks do that often. And so if instead of a single utility doing it for us, someone just had a complete set of instructions on how the darn boot-archive, path-to-inst and device tree worked together, then we might be able to reconfig the drive to the new hardware in say, less than a couple of minutes, it might not be so bad. Better yet, would be to script the process and have it as a command in the safeboot image. That's not there now, but something some of us are suggesting go in there in the future.

Lastly, once the system is installed, there's all the standard software that folks should stick on their system that isn't on there by default. There's the standard pathing for user shells that we should setup so commands are easily found too. There are also some nagging problems with devices that don't work well or at all. They don't impact the core Solaris kernel, but they may make the system unusuable (e.g. graphics is incorrectly sized, sound doesn't play, or some devices not functioning. Sometimes, an existing driver might actually work, only the vendor and device ID were not recognized in the OS database. Other times, the driver may be available as a free or commercial 3rd party just not on the install media. But their are quick ways to find basic drivers for network, audio, wifi and other components. These and other subjects will be topics in the next couple of blogs as I have time.

Looking back on the past couple of years and my experience with Solaris x86, I've gone through about 4 or 5 cycles where I've attempted to install latest current OS onto all my home and office boxes. At first, it was with mixed success, implying that a good fraction of the drivers were missing or so poor in performance and reliability that it made it unusable on the system. But lately, that's changed. Many do work for the low-end, budget systems in fact. Moreover, lately, the installations have been relatively easy. So it may not actually be so hard to achieve that high-level of fashion and popularity that being a Solaris install wiz, at least for a short while; most folks will still think installing Solaris is hard. But it's not something you need to share with everyone. And maybe you don't need to sit around all day eating cold pizza slices doing it to become a master at it. So you can have the sysadmin chic and still maintain the "girlish figure." The key is to enjoy this upcoming new year with Solaris, and sandbag a little when friends and colleague beg and grovel to have you install their laptop systems. Tell them you have a backlog and need more time. That might be true sometimes, but most of the time, I'd just take the fishing pole out and go fish a few hours while the install completes in about 30 minutes. I brought in a new custom fishing rod into the office recently. It's up on top of my locking bookshelf unit. Colleagues think it's just ornamental to go along with my fishing pic on the door. Hah!

Thursday Nov 16, 2006

What? Solaris Life Style?

Fishing Vacation Recap

Okay, so I haven't had time to blog in a while. Or fish for that matter. The exception was a short vacation back not too long ago to Vancouver to install a couple of new workstations in the vacation home up there, put in a second security camera and to figure out what was eating and extra 2kW-hrs/day in electricity besides the frige. I did manage to get some fishing in there, and a little scientific method of observing when the fish arrive relative to the tide, yielded big results. Here's a 10+kg salmon caught from shore just outside the house. Yeah, that's me in the picture. Took me 20 minutes to land the fish and two boats anchored just off shore to watch me bring it in. I wish I had a net with me, but I rarely bring one for fear of "Murphy's" Corollary - which states: If you bring a big net, you won't need it. But the fish looks like a Chinook on the outside. Black mouth, massive size, fiesty. And I got a kick cleaning it in the driveway. More than several of my neighbours (who own boats) drove by, hit the brakes, then backed up and lowered their passenger side power windows. "Ooooooh. N-I-C-E fish!" they all said, licking their chops. Amazingly, this salmon had no pink flesh. It's known as a "White Chinook" or "Ivory King." And up in B.C., there appear to be anywhere from 1% to several percent of the Chinook population that have the same genetic mutation where they lack the enzyme to breakdown and absorb beta-carotene. Hence, their flesh isn't pink. The flavour also is very sweet and mild, like a trout or char, and hardly like regular salmon. A delicacy in my book, for sure.

The two workstations I installed up there were systems for each bedroom. I found this killer store in Richmond, B.C., just off Bridgeport Rd., not to far west of the Home Depot/IKEA. It's called CanComputer and they have some good prices that compare well with the US. I may have mentioned some series of shops along the Bridgeport ICBC row (the section of Road where lots of auto-body repair shops are). Most are hole-in-the-wall shops with small store fronts, little retail display space, but large back rooms with inventory. CanComputer is like one of those shops, just a little fancier and cleaner. But they're all more mail order fulfillment businesses as opposed to Best Buy (which moved in last year, next door to the Future Shop at Landesdown Mall), which is over-priced anyway. Plus, the folks at CanComputer seem to know there specs a bit better on parts than the Future Shop or BestBuy folks. At least I was impressed that a sales person at CanComputer actually knew the difference in form factor for a x16 lane PCI-express slot versus AGP slot on an AMD socket 939 board and quiet power supplies with 20+4 versus 24+4 pins power rails. But neither BestBuy nor Future shop sell many computer parts - rather, they sell whole finished systems, so they aren't the kind of shops you'd expect true geeks/nerds to frequent.

I also had the privilege to bump into Nick Harrison at the Richmond, B.C. Costco. I was doing a run for DVD videos and on the lookout for Season 9 of Stargate SG-1, and met a semi-tall, medium build guy in the same video section looking at Sci-Fi serials. For folks who don't know, Nick Harrison is the guy who plays the Dark Knight in some of the Arthurian episode in last year's Stargate SG-1. He played roles as Jaffa in previous seasons as well. But it's just a kick to meet up with folks from popular Sci-Fi shows who are nice people and approachable. And with all the movies and TV shows they make up in Vancouver, that's another bonus for going up there. BTW, Season 9 of SG1 is out on DVD now. In the US, it's $32 at Costco.

Solaris Life Style?

So what's to do during those lazy late summer/early fall evenings in the great North? Well, for one thing, you can sit around and rip CD audio into mp3s to stuff them on a new, cheap, no-name MP3 player. I picked one up for fishing and roadtrips in California for around $50. It's a Truly MP310 in lime green with FM radio, colour display and 512MB of flash memory. I saw a recent sale at Fry's down in California on the same unit for about the same base price but $35 after rebate (curse short profanity to self...should've waited!) My sister always recommends waiting for these cheap electronic gizmos. "They'll get so cheap, you'll get 'em free with a Happy Meal in a year," she says.

Usually, I carry a trusty laptop with Linux on it, but as you all know, upgrading is a crap shoot, and sometimes, in the middle of an upgrade to a new kernel version, the Linux ABI story ain't the best and something go wrong. So my triple boot laptop has been booting Solaris x86-only in recent weeks/months. And it'll stay that way until I find some time to fix the install without hosing all my data on the existing slice. Well, I thought it was a bummer that I didn't have Linux with me, since, for things like CD-ripping and burning, it's been very handy and relatively easy to use. In fact, I believe the ability to rip and burn audio and mp3 CDs (as well as hook up digital cameras and other device support), has been critical to Linux's success. Without those utilities, there wouldn't have been the widespread adoption of Linux in the enterprise. In essence, Linux offered the Geeks something other Unices couldn't - LIFESTYLE.

Getting the powers-that-be who direct Solaris to view Multi-media support on Solaris, has been a long struggle for a lot of folks inside, and for many years. Senior/Upper management got the need for an Office Suite pretty quick. And Open Office/Star Office is just an awesome product. But office productivity doesn't equate to LIFESTYLE. A word processor isn't fun. You use it when a paper is due and you gotta get the darn thing out. You use a presentation package when customers are gonna show up the next day and you need to make something look pretty to sell them on some product or idea. And a spreadsheet... well, it's useful when creating PTA invoices to submit for reimbursement.

I thought Solaris was a long way from offering any Lifestyle, but I was wrong.

Stuck without operational Linux, I gave Solaris a try. And reading the man page on the cdrw(1) command, I quickly found out that it's possible to rip audio tracks and quite easily. For example, if you want to rip the 7th audio track off of any audio CD to a .wav file, then just do this:

    % /bin/cdrw -x -T wav 7 myaudiotrack.wav

So imagine what any shell scripter or Java programmer could do with a command line this simple... well, yes, it was easy enough to script this and rip and re-title every single song off of a CD and it's pretty fast too - probably less than 4 minutes to rip an hour of music into .wav format.

Once in .wav format, to convert it to .mp3, we can use any program. My preference is for Lame. And getting any version 3.9.x or later compiles cleanly and easily on Solaris x86 and runs fairly quickly, ripping at close to 6x playing speed for 128kbps bitrate. Usage is also straightforward. My ears aren't finicky so I don't need a higher bit rate. And if your cheap mp3 player is only 512 MB, the files are smaller than ripping using a higher bitrate.

  % /usr/local/bin/lame -h -b 128 --ta "Artist" --tt "Title" myaudiotrack.wav myaudiotrack.mp3

You may want to keep the .wav files around if you ever plan to burn a backup of your CD. But I usually delete them and just keep the .mp3 files. Some folks don't like the fact that .mp3s are lossy and noticeably lossy at 128kbps, so you can rip at a higher rate and store them which is still much smaller than the .wav files. I convert the .mp3 back to .wav files using mpg123 which is available freeware, and included on the Solaris 10 update 1 companion software distribution. If you've installed the companion software, then you'll find it in /opt/sfw/bin/mpg123. The command to convert songs back to .wav is:

    % /opt/sfw/bin/mpg123 --wav myaudiotrack.wav myaudiotrack.mp3

When you have all the tracks you want in .wav format, burning the audio CD is straightforward. Just use:

   % cdrw -a [track1.wav] track2.wav ....] trackN.wav]

So all this playing around with audio clips made me reconsider if Solaris can yield some semblance of LIFESTYLE. Yes. I think it can. The real question though is whether it could load songs onto the el cheapo MP3 player which is supposedly a usb-storage device. Funny thing is that the manufacturer claims that one can use Linux 2.2 and later with this player. But some users have had issues with kernel 2.6 and the usb port with this device. I had some issues with Fedora Core 3 at home, which has been upgraded a few times. There appeared to be an "Error USB Port 1" on Linux in the log files. I didn't expect much on Open Solaris Nevada build 48 if Linux couldn't even see this.

So on to Solaris. And as expected, nothing happened. Well there was some output in the /var/adm/message file on Solaris, but the usb-storage didn't mount. Yes, I was a little disappointed, until I got an idea to contact some of our Beijing driver folks working on USB. I got my hands on an early version of USB driver and after installing, I tried plugging the device in and, WOW! The device mounted and was found in /media. I found out that the MP3 player only supports USB 1.1, so it's a slow dog to upload songs. But after a couple of hours of chugging along, I managed to load up some oldies and goodies for my exceptional entertainment-on-the-go. I've checked recently with build 51 of Solaris Nevada, and yes, the new USB is in there, so soak up the Lifestyle folks. And if you have some cool scripts or Java GUI front end for the command line utils to rip and process audio clips, let me know. I would upload mine to Java.NET, except they're pretty embarrassing right now. Maybe later after code clean-up.

[disclaimer: Please use information here only to create -legal- archives of music you own. Don't be a spoil-sport and steal music. Support your favourite musicians and their labels].

Tuesday Mar 28, 2006

Adding a pxeboot driver to Solaris 10 01/06 x86

So a partner came in last month with a network driver for a blade system. They developed it using a whitebox PC running Solaris with a reference PCI card. But they never officially tested it on the blade system the driver was intended for because they couldn't get the driver on the blade... at least not conveniently. The problem is that while they could power the blade inside a special case, without a full chassis with optional shared optical drives, it was pretty tough installing the OS because the only way to install it on the blade was to do a network install. But since this was a new network driver, it wasn't in the installation OS.

The BIOS and the NIC supported PXE boot and while Solaris could begin the installation, it would stop and exit to a shell prompt the moment the PXE handed control over to the Solaris installer. Mind you, we're talking about new boot in the Solaris 10 Update 1 (aka Solaris 10 01/06) which now uses grub bootloader. But this was a classic example of the Chicken versus Egg problem.

The partner had a good working knowledge of PXE boot and they followed the docs.sun.com document and others they found on the net. When PXE booting a PC, the install client receives a number of options from the DHCP server that direct it to fetch a runnable ramdisk image which then bootstraps the rest of the system.

In the case of Solars 10 1/06, the boot client receives DHCP option entries that tell it where to download what a Solaris x86.miniroot root image, which is about a 50+MB bootable ramdisk image. The PXE BIOS handles the entire network transfer from the server, then unpacks the image and boots it. Control of the network is then handed over by PXE to the Solaris miniroot.

Now, if a necessary network driver isn't in the Solaris x86 miniroot at this point, well, the installation stops, prints an error and exits to a shell prompt. This was pretty frustrating for our partner who knew they needed to get their driver into the installer image but didn't know how. Solaris 10 x86 update 1 came to the rescue. The new boot architecture has added several commands that simplify the packing and unpacking of x86 miniroots. To add a driver, one simply unpacks the miniroot, adds the driver and repacks it.

To do the first step, unpacking the miniroot, a user can run the root_archive(1M) command on the install server (presumably it's running Solaris 10 x86 1/06 - update 1 - since older versions and SPARC do not have this utility):

  # /boot/solaris/bin/root_archive unpack [pxeserverpath]/boot/x86.miniroot [path-to-unpacked-dir]

Once the miniroot is unpacked, one copies the 32-bit driver binary and driver .conf file to the [path-to-unpacked-dir]/kernel/drv directory, and then runs the /usr/sbin/add_drv command with the right PCI IDs and the right permissions against the unpacked miniroot directory. E.g.

  # add_drv -b [path-to-unpacked-dir] -n -v -m '\* 0600 root sys' -i "[device ids]" [mydrivername]

where, [mydrivername] is the name of the binary just copied to the unpacked miniroot ./kernel/drv directory, and the [device ids] is a string list of PCI device IDs that might look like:

  '"pci1a44,9043" "pci1a44,9065" "pci1a44,9106" "pci1a44,9053"'

Note that the -b flag allows you to set the root path where to apply the driver add operation, which is pointing to the unpacked miniroot. Note also that the x86.miniroot is 32-bit and only supports 32-bit drivers.

The last thing to do is to repack the miniroot using the same command, but specifying "pack" as the keyword. But before we do that, I like to make a copy of the original x86.miniroot in the same boot directory, and then run the root_archive(1M) command:

  # /boot/solaris/bin/root_archive pack [pxeserverpath]/boot/x86.miniroot [path-to-unpacked-dir]

If the PXE boot server was working before, the client should be able to boot and load the x86.miniroot and complete a normal Solaris 10 1/06 installation. The installation will bring up a menu of install choices; the default is to perform an interactive install. When doing an interactive install, the installer offers a choice of auto-reboot, or manual reboot. Choose the manual reboot, because while the above steps get the driver into the miniroot, they don't do anything to install the missing network driver onto the final client system. So we select manual reboot, to give ourselves a chance to copy over the drivers and run the add_drv command on the final client disk image before rebooting. Otherwise, the newly installed system, still won't have a network driver.

So where do we put the driver binaries so the install client can copy then over? One way is to copy the one from the miniroot's /kernel/drv that's already there. This works if the system is only 32-bit and will only run in this mode. However, if the system is x64 capable, we didn't copy the 64-bit driver binary into the miniroot's /kernel/drv/amd64 directory and that's because the miniroot is only 32-bit and doesn't have a ./kernel/drv/amd64 directory. We could have created that directory in earlier steps, but that's not necessary. In fact, during the Solaris PXE boot installation, the client mounts the install server directories via NFS. Thus it's easy enough, before rebooting, to simply just put the driver files in the install server's exported PXE boot directory, especially inside the jumpstart directory where the client already mounts them during the install. I usually just put a ./jumpstart/drv subdirectory inside the server's exported directories. This will be mounted on the install client as /tmp/install_config usually. If you can't find it, use the df(1M) command to see a list of filesystems and their mount points.

So now we can persist the drivers to the install client. The installer miniroot mounts the client's hard drive usually at /a. So it should be straightforward to copy the binaries and .conf over to /a/kernel/drv and /a/kernel/drv/amd64. (Note: there's usually just one .conf file for the driver that goes in the /a/kernel/drv - which is used by both 32- and 64-bit binaries.) The we run the add_drv(1M) command again, but for the -b option, we point to the /a directory.

If this is a custom, hands-off jumpstart installation, it's possible to modify the post-installation finish script to copy the driver (both 32- and 64-bit binaries and .conf) over to the final system disk and also run the add_drv(1M) command so all systems of the same type get the same new network driver without manual intervention.

Our partner with the driver problem was pretty pleased that this worked, and surprised that it wasn't all that hard to do. We're hoping to make it even easier in future versions to add drivers into the installer and even create bootable CDs and DVDs and burn them all in Solaris.

Tuesday Mar 21, 2006

The cheap Solaris 64-bit box pending rebates

I was on a quest 6 weeks ago to see how cheaply I could build a 64-bit Solaris 10 box that I could use in the office, and gave myself a 2 week period to hunt for deals. The quest wasn't all that hard, and what I've ended up with wasn't all that bad of a system for office use.

At the core is an ECS K8M800-M2 motherboard and AMD Sempron-64 cpu. I got this on sale for just $69 together at Fry's on a one-day sale. I had to be careful that they didn't switch a 32-bit BABOX processor on me, so I insisted they honour their ad for a 32- or 64-bit mode cpu which they did, and I got a 2600+ 64-bit BXBOX retail cpu (the box actually has 64-bit on the lower right front corner). It's socket 754 which is probably going out of date soon, but for a functional system that I don't plan to upgrade, it'll get the job done and hold its own for a few years. The board has on-board LAN, Audio, and Graphics, so hopefully, this would make a pretty cheap system. And best of all, no rebates required.

Next was the case. I found the Antec 1650B black case w 350W power-supply on sale for $59.99 with $30 rebate. I'm not fond of rebates, but if I see a good deal and the rebate is from the manufacturer, like Antec, I usually go for it. Plus, this is a fairly quiet case, with easy-install and tool-less takedown. I have an identical white case in the office which ran my old ECS K7VTA3 mobo, and it was pretty reliable and quiet.

A few days after I found the case, I saw an ad for 200 GB WD SATA drive retail kit with cable for just $49.99 after a $30 + $20 dual rebate. And about the same time, I also found a dual stick 1GB of OCZ DDR400 (2 x 512MB) for just $75 with $25 rebate which ain't bad for branded memory. And for optical drive, I found a black retail NEC 3550A 16x DVD burner online for $39 w/ free 3day shipping.

A few days later, I put the box together, and installed Solaris 10 1/06 on the system. First boot wasn't very cheerful. The installer graphics couldn't recognize the onboard Unichrome Pro graphics, and defaulted to text console install, which did complete, but still couldn't get the Xorg to recognize the Unichrome Pro. The SATA controller for the mobo apparently runs the disks in IDE legacy mode so boot and install were transparent. I didn't have to turn on any switches in BIOS even. But I was stuck with no X-graphics with the Unichrome Pro, so I exercised the AGP option, since the micro-ATX board DOES include an AGP 4x/8x slot.

I found a refurbed Diamond S60 board for sale at a local surplus. It has AGP 2x/4x I think and the ATI Radeon 7000 chipset, which is well supported on Solaris. It was a 32MB card and cost $19. I knew I could get 64 MB cards these days for that much brand-new, but in a pinch it worked and I had a working system. Plus, with the newer cards sucking so much power and competing with the CPU, the older card with slower GPU seemed like a wise choice for the 350W power supply.

I added extra USB 12-in-1 flash reader for $7.99 later, and an $8 floppy drive a bit later with a $3.99 round 10" IDE floppy cable for better airflow. I also bought a $3.99 12" IDE ATAPI cable for the DVD burner as well.

Altogether, the cost for this system will come to about $300 after tax and if all the rebates come through. It's not great, but another decent deal for a 1GB sytem with 200 GB disk, universal usb reader and floppy, network, and audio.

Getting Solaris up and running with graphics wasn't hard, especially with the extra AGP graphics card plugged in, but the on-board NIC requires a 3rd party VIA Rhine-III driver. Luckily Murayama's got free Solaris drivers on his site and so I was able to download and compile the driver and install it. A new version 2.0.1 of the rh driver is available and runs well on a number of my systems. It's based on the new Solaris GLD framework (generic lan driver). If folks haven't compiled drivers on Solaris 10 Update 1 (i.e. 01/06 release), there's a slight bug on line 206 of the /usr/include/sys/ddi_implfuncs.h header file. Just comment that line out and if you make clean, then make; make install in the driver build directory, it should just work. Murayama also includes pre-built binaries as part of his distro so a make; make install doesn't actually rebuild the objects unless you 'make clean' first.

In booting this system, a psrinfo -pv reports the CPU is AMD Hammer Family processor - Model Unknown and isainfo reports that the system by default boots into 64-bit mode. I've had the system up for close to a month now with no unscheduled downtime. It's been a real champ for an office workstation and even for development. With home directories mounted remotely and most of the building switches defaulting still to 100Base-T, the system was more than capable of working well and quickly, even with compiles. I'd like to see a Unichrome Pro graphics driver for Solaris x86/Xorg soon. If that happens and works stably at 1600 x 1200 pixels on the 21" LCD flat panel display, then I can probably pull the AGP card out and save a few watts and just keep working.

Friday Nov 04, 2005

Silent and Cool - Home Upgrades to Solaris x86

Solaris running on a mini-ITX fanless system inside a shoebox case is quiet and stable.

It's been a long time since I blogged. I guess I just got tired of blogging. And plus, I got caught up in other things in life like my first kid starting Kindergarten and all the other things that happen - like getting to know the school, the teachers, and the other parents and getting involved in the PTA. I also switched groups internally. I'm working with Solaris x86 and hardware vendors. It's a heck of a learning experience and lots of fun.

Speaking of Solaris x86, the momentum has really taken off. Someone mentioned that we hit over 3 Million downloads the other day on S10 alone. Wow. Not bad for GA - General Availability - in March of this year. But sadly, I've somewhat of a hypocrite. Sure, my work laptop has Solaris x86 on it, but it's triple boot, and to get wireless networking in airports, I'm still booting Linux more often than not. And at home, I'm still mostly a Linux shop, even if I do have Solaris on a couple of boxes.

But that time has come every few years to upgrade. My last attempt failed last year around the time Fedora Core 3 released. I had the FC2 CDs and was excited about upgrading, frankly because some of us geeks love slapping our own boxes together :-).

I saw a sale at the local Fry's and bought 3 low-power VIA c3 mini ITX boards for about $89 each back then. And shopping online, I found a sale on some Inwin low-profile bookpc small form factor cases for $40 with free shipping. I was pretty excited waiting for the parts to arrive and when they came in, I assembled my first box and slapped the FC2 install CD into the drive, booted up and then got a big REJECTION when FC2 quit the install telling me that my hardware wasn't supported. I tried some Redhat Enterprise and it too wasn't supported, and then some Solaris 9 CDs and that wasn't supported. And that wasn't the only disappointment, the 60mm P/S fan inside the case was a 5700RPM noisy bugger that screamed at close to 40dB. It was inward facing and already muffled by the case, but still, it was pretty loud, and at night, when the kids went gone to bed, it made my house sound like a data center.

I did manage to solder two 30 Ohm resistors onto the P/S fan wire and it slowed it down a little but it still whined loudly, albeit, no longer loud enough to make me want to take a sledgehammer and kill it. I knew I could go ahead and install FC1 which I already have running in my bedroom in a quiet PC. But, but I decided to download FC3 which folks reported worked with VIA c3 systems. The install was stable, but FC3 turned out to be quite unstable in itself, locking up constantly due to problems in the graphics. I tried the same board in different cases, different power supplies, with new DIMMs, and still had instabilities that would lock up the system even though the text mode worked fine.

I shelved those boards until recently, when I bought a couple of new Antec Aria cases. Initially, I ordered 2 for work, to do some testing in low power Solaris x86 systems and compare them to Linux. The cases were so quiet that I went out and bought two for myself. I also tried out FC4, but it turned out to be somewhat disappointing and not really stable either. And by instability, I mean that it would lock up after 20 hrs of uptime. I've downloaded all the upgrades for both FC3 and FC4 and it hasn't improved the stability, even though I've spent quite a few sleepness nights swapping boxes, power supplies, and memory. And with certain brands of ps/2 KVM switches, the Linux 2.6 kernel seems to hiccup and inject a button 2 or 3 event with using scroll mice.

But enter Solaris 10 x86. The installation is still buggy and the kdmconfig core dumps when trying to bring the XFree86 installer up. The failure converts into a pastel screen text console with funky colors for text when it should be black on white. There's also a problem with reverse text visibility. But the text is actually there and the install proceeds. It only happens with certain graphics chips and BIOS sets is what I'm told, like the VIA chipsets using Unichrome or formerly called CastleRock AGP. Prior to December 2004, Solaris had never been able to recognize VIA c3 as a valid x86 cpu and would crap out during the secondary boot. Since build 69 back late last year/early this year, the kernel team said they did a put back that fixed the CPU recognition issue. But this has really opened up the market for low-power OEMs like Igologic.COM to supply the Jbox running Solaris.

Ironically, I hadn't actually tried using the 3/05 GA release bits for s10 to test my mini-ITX boards back in March. I did test using some CDs that someone had burned for me with "S10x86 GA" on the labels, Disk 1 - 4, but when they installed, they crapped out on the secondary boot with the "Unsupported architecture" error and I never bothered to follow up with my mini-ITX boards being too busy. After all, there was a workaround for the bug. Prior to S10, I had been using a hack to replace the GenuineIntel recognition string in the kernel binary and on the ISO install image with CentaurHauls (some funky name for the family of VIA c3 processors). I was thinking that the S10 bits only supported some of the cores like the Nehemiah but not Ezra or Samuel cores.

But 7 months later, with a couple of test systems at work I want to install, I looked into the CPU recognition problem and behold, the old media I had didn't actually have S10 GA bits, but build 67 bits. So I went home and actually tried my own mini-ITX systems with S10 GA and it installed. With the exception of the kdmconfig errors during initial install, the process was straight forward and I had a graphical workstation up and running in an hour or so. Xorg with JDS boots and configures itself for 1280x1024 24 bit graphics. The USB storage driver works too, as does the camera tool. One needs to restart the volume manager daemon (/etc/init.d/volmgt [stop|start] ) which is a legacy service daemon not folded yet under Solaris 10's new Service Management Framework (SMF).

I had to go online using another system to obtain VIA Rhine Ethernet network drivers and the VT 8235/8xxx audio drivers as well. But both sources of drivers had recent tests by their authors on Solaris 10 and they provided portable driver build srcs that were just 100kB to download and supported Solaris 8, 9, and 10 versions. that easily fits on a USB Jump Drive and within a few minutes, I had audio and network up.

The build size and archive for these drivers is pretty incredible when you think how big a Linux cross-platform kernel driver build environment has to be, and the compatibility issues between kernel 2.4 and 2.6 versions. I've been running Linux since the early 90's when I saw my first set of CDs walking along the streets of Akihabara, Tokyo. They were Walnut Creek CDs if I recall, and they still distribute Slackware, if I'm not mistaken. But the kernel source build environment has really bloated since then, and for companies trying to support -for-profit- proprietary software, such build systems and lack of ABI compatibility is tough. I've worked with some Linux IHVs now that are porting to Solaris x86 because their minimum support source base was 30+ Gigabytes! This is for all the cross-compilers, GCC versions, and kernel source versions for each flavour of Linux, be it SuSe, RedHat, Fedora, and their corresponding updates, and 32-bit versus AMD-64bit. The comparative build and install on S10 is well less than one GB.

The Antec Aria case is around $100, and has front USB, Firewire, and 7-in-1 USB flash reader. Older VIA EPIA-800 mini-ITX motherboard with cpu and fan will cost around $110 today and probably not have all the pin-outs you need to hook up the Mobo. But if you're like me and look out for hardware deals on quiet PC components, you can core a sale on boards like I did for $82.95 at Fry's a few months ago on clearance for VIA Eden 600MHz ME6000 fanless boards which as firewire and compliant USB pinouts. A 160GB disk will run you about $69 after rebates, and an NEC or Lite-on DVD burner 16x with bonus floppy drive OEM might run you around $50 if you catch those online, one-day sales like at NewEgg. A 512MB stick of DDR can run between $30 - $90. Since these chipsets don't support ECC, I've learned that it's probably better to get higher quality name-brand memory. But for around $400, you can build a pretty quiet Solaris 10 x86 system that won't break any speed records or nuke aliens in gaming, but it works great for vpn, email, web surfing, and office productivity. And it's beautifully quiet.

Thursday Jan 27, 2005

Spam zombies and port scans - to log or not to log

Not much happened back in December except it was cold and rainy all along the West Coast. But weather aside, I was ignorant and blissful about the security of my networks both here and up at my other place in British Columbia. I did have a chance to drive north with the family and score on some Boxing Day sales in the great, dry and friendly North, but the visit was simply too short and I only had two opportunities to enjoy great dimsum. I've been back in California for quite some time now and quite busy. However, prior to my return, I setup my B.C. WiFi network to do more logging - both on the internal interfaces as well as the external WAN interface. And I had my reasons.

It was exactly a year ago that the big ISP in B.C. offered broadband DSL along Highway 99 up to Whistler, and I was one of the first on that leg to subscribe. In just that year, I've seen some tightening up of packet filtering on the ISP's network. Within about a month after DSL became available, apparently, I picked up intrusion attempts by at least three other compromised systems from neighbours. I sent out some email to the home owners up there to be on the lookout for unusually heavy network activity on their routers when they weren't actively using their systems. And I wasn't the only one that noticed. About 6 months into our new found bandwidth, the ISP decided to shutdown free flow of port 25 SMTP traffic from any subscriber except through their mail routers.

Such action by my ISP was annoying, but easily circumvented by tunnelling packets through a virtual private channel back to my mail server back in California, so my problems were solved, and I still retained some autonomy and privacy. But, I'm sure that for quite a few customers, including some of my neighbours up there, the ISPs actions caused some grief. You see, for some of my neighbours, a significant fraction of their emails began to bounce and were no longer getting to their intended and legitimate recipients. Mail was bouncing due to the ISP's mail servers getting onto DNS blacklists as primary sources of SPAM. And the reason why the ISP's servers got blacklisted was that the spammers adapted to the block on packets destined to port 25 on non-ISP servers; they decided simply to route email through the ISP's mail gateways. And to avoid the ISP from tracking down all the compromised systems, the spammers didn't ust use a few spam-running-zombies, these folks compromised hundreds of systems and had each one send just a few thousand emails and then stop after a couple of days, until the next campaign. This caused half of the ISP's mail servers to get onto some of the major DNS blacklist servers out there, and I would guess their tech support guys had to field a lot of calls from folks that ended up with rejected emails and needed to switch SMTP gateways. Three or four out of the half dozen ISP's mail servers in the lower mainland B.C. and Alberta are currently or were as of a few weeks ago, on the top world's DNS blacklists for sending too much spam. In fact, I think more than 90% of email coming out of their network is spam, at least I block about 80 to 90 spams a day from them with no abatement and 99+% of that is spam. And while I've configured my mail server at home to block spams and return a polite spam Error 550 messag, the ISPs around the world that route spam emails often just seem to ignore, or worse, forward the problem, as opposed to aggressively dealing with the situation and solving it.

A clear example was a case of rejected spam which I tracked was originating from a poor guy in southern California, who evidently, suffered a fatal disk crash after I contacted him and told him about the problem. He ended up having to re-format and install his operating environment. How I found out his system was compromised and a spam zombie was quite a coincidence. A month and a half ago, during a 30 minute period, I received over 1000 emails from about 5 MTAs worldwide that were bouncing an undeliverable spam to me, the apparent sender. Fortunately for me, 4 of those 5 mail servers included the message with full headers, and clearly, I could tell that the first hops and last hops were not from my IP address domain. But the ISPs should have easily figured that out and just killed or dropped the email or simply denied mail routing because the mail headers and addresses were so obviously mismatched. But as I said, many ISPs are just sloppy about mail filtering and don't bother. So despite me obviously not being the true sender of the spam, these ISPs just let me have the flood of bounced emails. In fact, one ISP's automated SPAM fighting machine apparently recognized the 250+ emails it got as spam, but then decided to reply to the faked Sender address with some legal-mumbo-jumbo about abuse of terms of service. Geez. I felt like half these ISPs were just playing dumb and arrogant. Clearly, their own header information encapsulated within the email indicated it was a spam and the sender domain and MTA IP address had huge mismatches.

To stop the flood for the next few hours, I decided to simply block all emails to that address and send immediate errors messages that explained that this address was not valid. The campaign did only lasted about 2 hours, and then the numbers of messages subsided after several thousand bounces. How many were actually delivered, I don't know. It did make me think.

But the lucky coincidence for me was that all the spoofed emails were using an email alias that I publish to the network for just my fishing msgboards. And by inspecting the headers in some of the bounced emails, I quickly found a common point of origin from an IP address in southern California. And the two pieces of information led me to check my web server access logs, and I did get a match, plus a bonus piece of info: a cookie ID. This cookie is something I plant in my web pages that can help identify unique sessions, especially identifying HTTP connections for logins.

This allowed me to identify the actual user and again, luckily, he had contacted me in the past and left an email address. Unfortunately, he was a skilful angler, but not a big IT technician, and so he wasn't sure beyond running standard anti-virus software how he could stop being this spam zombie. And unfortunately for him, I guess the folks using his system for a zombie were finished and didn't want many traces of their activity. Within just a day after I notified him, his computer disk crashed and all data was lost. After a week of silence, he emailed me back and told me about the crash, and the subsequent re-format and re-install of his entire system.

All this bad network activity in the past month or so spurred me to turn on aggressive logging on most of my home server and router systems. So just two weeks ago, I started to get a rash of panic emails from my router up in B.C. Evidently, pings of death were being detected and I had set the system up to email me immediately. Again, like the spam incident I was getting copious emails, this time not as quickly, but they were averaging many per minute and they indicated that the attacks were coming from 7 separate networks in at least 3 different countries. For two days, the router logs were arriving in my mailbox here in California almost once every 30 seconds. I wrote the ISPs to politely forward abuse emails to the right folks in their network to stop the attacks on my hosts. Most did have an automatic mail responder, but only the Germans sent back a personal response to my inquiries and told me that they have identified the host and have forwarded the headers and logs to those in charge of that subnet for investigation. After 3 days, and a few megabytes of logs and emails, the pings of death and port scans finally stopped up in Canada. I'm not sure if rebooting the router and getting a new IP address assigned was the trick or if the campaign just stopped.

Being the curious kinda person I am, I couldn't just be satisfied with the status quo, so I decided to turn on aggressive packet filtering and logging on my local systems here in California. I have two servers that run 24/7 and have open interfaces to the internet. I do have firewalls turned on, but I was not logging the packet rejections or denials. So for the last few days, I decided to turn them on and observe. Just between 2:45am last night and 8am this morning, I rejected about 120 attempts on my mail/web server, and about 40 attempts on my NAT firewall box. I have funky ICMP packet requests that don't look like pings of death. I have strange UDP and TCP attempts at really weird high port numbers that don't conform to any service or standard, and by far, the 90+% of denied packets are port scans for 139 and 445 NTFS file share UDP ports. And that was just in a little over 5 hours. It's incredible to me just how many scripts set off by hackers there are out there, and how many unique attempts occur to gain unauthorized remote entry onto a system there are. The costs must be staggering for folks with systems less robust and less protected against these hackers. But just having that knowledge itself can be pretty depressing, especially seeing how it means we need to be ever more vigilant against intrusion. It's almost enough to turn off the logging, save some disk space, and just live in ignorant bliss for a (short) while.

Monday Nov 15, 2004

JDDAC at the Romberg Tiburon Center - SF Bay Estuary Monitoring

Well, how often does high-tech work in large scale enterprise computing fall right into a fisherman's habit? I had a nice conference call with a bunch of University and Corporate technologist folks yesterday afternoon about a project that needs a world class software architecture to provide real-time or near real-time data about SF Bay water quality and monitoring. The idea is to get data every 1 - 10 minutes from a wireless sensor grid that monitors all the estuary waters, and even middle and outside of the Bay and provide that data accurately and quickly to all users. Such a data would include real time lookups of conditions, as well as historical data monitoring for scientists, researchers and policy makers.

Sun and other companies started an initiative some time ago called Java Distributed Data Acquisition and Control (JDDAC). And immediately, one can see a strong business case for JDDAC. The challenge today in manufacturing, for example, is retooling costs for high-margin custom manufacturing. In other words, businesses can charge more money for some amount of work if they do custom work. But the costs of retooling are prohibitive and so customers tend not to order custom unless really necessary because it does cost so much more and manufacturers rarely provide discounts on any custom job unless one makes a very large order of things. The purpose of JDDAC was to standardize remote sensing and control technology so that businesses could retool and customize their manufacturing line to do more custom work at lower cost, thus meeting both customer demand and lowering business cost.

But then recently, some SF State U. researchers at the Romberg Tiburon Center that do Marine estuary research got wind of this initiative and quickly connected the dots. They've been funded by various gov't agencies to monitor Bay Quality. One of their major data consumers is NOAA. And some of their data is used in policy making for all sorts of things, from water diversion upstream, to quality standards evaluation of habitat for fish and wildlife. My details are only superficial and I'm learning more about this as I go along. But it sounds like a lot of fun.

I think I heard that they got a grant to design and build a new sensor grid architecture to monitor water quality. One of the design goals will be to standardize on software and hardware interfaces so a grid could be quickly and efficiently deployed anywhere (even exported to other sites around the world) to monitor many types of conditions - e.g. water clarity/turbidity, bio-fouling, mineral and other elemental concentrations for CO2, O2, pollutants, salinity, etc. etc. The system would need to scale as well. Starting with just a dozen sensor stations with sub-grids of sensors, the system needs to support thousands or more and do this in real-time. With such a plethora of sensors, having a standard interface and software to do data acquisition and control are vital if they were going to succeed.

Today, they have hardwired sensors mounted on some big concrete pilings along some shorelines. These are then connected phyiscally to computers and every 1 - 5 minutes, water quality data are gathered and store on disk to be processed later. It is non-real-time, and requires considerable human attention. The new system they want to build would be automated, wireless, and use standard sensor interfaces like JDDAC to collect data and control sensors. For example, one of the problems of taking, say turbidity measurements (murkiness of the water) is that algae and other crustaceans can foul the intake sensors. Their solution has been to put yet another remote control hardwired blower/pump around the sensor grid to clear the fouling prior to each measurement. More wires, more maintenance, more downtime. Ideally, by having a wireless and standard control interface, one control can proxy commands for another (much like USB or Firewire devices... you can chain them together). But there are obvious advantages to this technology for monitoring water quality, and when I brought up the concept of fish census, these professors all understood the challenge of figuring out how many fish are actually around in the water, data DFG/FGC need to assess stocks.

As an avid angler inside Bay waters concerned about Bay Water quality, I was delighted to see such initiatives for real data collection be funded and proceed at such a rapid pace. I was also delighted after a senior colleague of mine that leads the specification team invited me to become one of the participants for the software and network architecture. When the SFSU professors spoke about pier pilings and tidal currents and bio-fouling and piers, I could picture exactly what they were dealing with having spent some years now fishing these areas. And my colleague grinned because he knew that for this particular project, I was perhaps as eager to support this initiative as I was qualified from both the software and marine aspect.

And interestingly, word has come down from the Fish & Game pipeline that some folks down south in Monterey may be interested in a sensor station. Funding has occurred and I heard they are building a new Pier near the Moss Landing Jetty. This pier may only be primarily used for research vessel mooring and Monterey Bay science projects is what I heard, but that's early news and more details need to be researched. I need to contact some folks at CSU Monterey Bay to see if this is one of their projects and to ask if the might want to collaborate. I attended a great little symposium at CSUMB back in July or August and saw a lot of research posters from collaborative Universitys all over the West Coast and US. The symposium was sponsored by the NOAA (Nat'l Oceanic and Atmospheric Administration).

Who knew just fishing the Bay from pier and shore and along our estuary waters would have such beneficial and synergistic consequences? More to come...

Friday Nov 05, 2004

BBBQH2 for Techies

Big Barbeque HowTo for Techies

Too busy to blog!

I stopped blogging for quite a few weeks now. Just too busy to even get my head out of work mostly. I was one of the lead organizers for an annual internal technical training and conference and headed up IT, registration, audio/visual and food for the week-long event. Our group of about 200 people worldwide has undergone some major changes in the last few months. A complete re-organization. All new VP, Sr. Director, and Director that I report to now. I'm in a new position too, doing similar techie stuff, but under a slightly different charter and for slightly different vertical market segments. Morale was pretty mixed at the beginning. But people seem to have cheered up after the conference where they all had a chance to enjoy a few cold drinks and some good food. I must say that the week long event couldn't have been a more perfect venue. We had speakers like Andy Bechtolscheim, Scott, and Jonathan do keynotes for us. And they didn't disappoint. I felt it a personal duty to serve food that was at least worthy of such a great program.

Hosting a week long event can be tough. We had great help from our organizing committee. Content was led by Matthias, a German colleague out of Sun's Walldorf office. And logistics for budget and travel were handled by logistics expert, Don. He pretty much shaved travel, lodging, and ground transportation costs down to around than $1000 per person for the roughly half the participants coming from 13 other countries and 5 major areas around the US. As we all know, budgets can be tight, and directors are given very limited dollars for group team events these days. Times were more encouraging this quarter because those funds were more available. But still, the overarching directive was still to watch our costs. Still, I actually had the opportunity to lead 2 major events - a BBQ and a Luncheon and act as primary cook for a third (Fajitas & 'Ritas). The Fajitas and Margaritas party and the luncheon were employee funded with minor support from the company, while the BBQ in the park was picked up completely by the company. The costs? Well, the Wednesday BBQ for 160 people cost just under $5/person. The Thursday Fajitas and 'Ritas (Margaritas) Party for 69 people cost about $5 - $10 each depending on whether a participant imbibed, and the Friday luncheon for 180 was just $3/person.

Catering to my Colleagues

First of all, I love to cook for people and I love to grill. It goes back to my academic history. I was pretty active in student societies back at Cal Berkeley. In my senior and 1st grad school years, I was BBQ'ing almost 3 times a week for various student groups. These were mostly morale/team building events that were revenue neutral. But the joke in the College of Engineering was that I was the master of porous media Heat Transfer - the porous media being charcoal briquettes, real wood, or fake crushed volcanic stuff over a propane fueled flame. There was just something about grilling outdoors that put me in touch with my primal roots. Beat the tom-tom drum. Sing Kumbaya kinda stuff. Bond with nature, yadi yadi yadi.

Wednesday Afternoon BBQ

Getting back to the Wednesday BBQ. There are lots of ways to cater a large outdoor BBQ party, but quality and quantity can vary drastically depending on the help, menu and venue. And I can't overlook good help. Some of our group admins have tremendous logistical experience with events and they showed up and helped out. Our of our stars, Kim, is 8 months pregnant (almost 9 now) and she showed up early with 80lbs of ice! We also had 3 engineers show up early and post signs along the roads and park to guide folks to our destination, plus setup the chow lines and place all the food for optimum parallel processing of hungry eaters as they came through. The Directors showed up and ran all the slice-n-dice food prep operations for hors d'oeuvres and other items. It was real team work.

But for the best help, my recommendation would be tough for anyone else to follow, because I would suggest that you marry a spouse as helpful as my wife. She really pitched in with the shopping and the food prep. We were both up late the night before, first shopping and then prepping and marinating, and then up again at 4:30 am in the morning before the BBQ to prep some more, cook some of the dishes and load up the vehicles, etc.

BBQ Risk Management

The way I organize a BBQ is all about risk management. There's environmental risk, like the weather not cooperating. And there is operational risk. Certain operations can be delegated, like setup and cleanup. But some operations, like procurement, must be controlled by and given to a single party or we risk having the standard asyncronous potluck syndrome - too much beer, chips, cake and spoons, but no plates, forks, diet beverages and main dishes. Then there's the health risk. One challenge here is sanitation. Another challenge is storage. How many folks can handle literally 1000 lbs of liquid and then deliver it the day of the event? What about a hundred pounds of raw meat? How many folks have food-service experience? While the risk of e.coli and salmonella contamination are fairly low, with a company event, 200 high-IQ engineers could be made pretty unproductive for quite some time if some outbreak should occur. Having worked in a restaurant for 4.5 years and done lots of procurement, usually, that's a job I take personal responsibility for. This leaves Menu and Venue left as action items for others to decide.

A Possible Menu

Planning an event is almost like doing Product Life Cycle (PLC) Management, only the time frame is a lot shorter. You have an approval committee that has managers and architects, and multiple people are tasked with action items (in our case - a BBQ Event) and we draft half-pagers and one-pagers outlining our proposed Architecture or Menu. Cost/Benefit ratios are analyzed, small separate committees and product teams (cTeams and pTeams) form around separate tasks and we set milestones. As milestones come and go, we track progress, and finally, come GA, people pull all nighters to make sure stuff ships on time. In our case, it all started with my proposed menu. Since I took responsibility for procurement, the group thought it only appropriate to give me the privilege of doing the menu (not to mention it would save a lot of hassles all around to just have me do it.)

For openers, I was planning on smoke salmon on french baguette slices with olive oil, pickled capers and dill weed. A classic hors d'oeuvre, but somewhat labour intensive. Others suggested that we simplify and modify this and provide individual bags of variety Chips - a la Frito Lay-style. We thought about this and it was decided that we do both. The Directors we given the assignment to fabricate the smoked salmon hors d'oeuvres and they did an okay job, although they skimped on the smoked salmon and only used up one pack of salmon and left about 2 baguettes. The next step was to provide some first course (Prima Piatti) options. Usually, in the states, this means salad. We decided to offer a tossed mixed salad (insalate miste) but with a Ranch dressing and small tiny olive tomatoes plus optional Potato salad. Both are extremely affordable and come mostly pre-packaged off-the-shelf. Next was to provide a Carbohydrate option prior to or with the main (second) course. We considered the sensitivity to all the Vegetarians and came up with two optional items. Garlic Bread Forte (strong garlic bread - Bamm!! with like lots o' Garlic), sweet dinner rolls (for folks with the Vampire retro virus - intolerant of garlic), steamed Basmati rice seasoned with Tumeric, Saffron, Cardamom and Raisins, and lastly, spaghetti with a simple marinara sauce. The main course would be an assortment of grilled boneless beef rib steak and boneless dry rub seared chicken thigh meet, italian sausage and Cajun style Hot Links and Portabella Mushroom Burgers. For desert, we planned for any bulk bags of seasonal fruit like apples and oranges. And finally, for drinks, we went with a full non-alcoholic beverage list of all individually bottled or canned sodas, diet sodas, sparkling water and plain bottled water sans gas.

Note on Vegetarians/Vegans

Vegetarians are categorized as folks who do not eat meat, but may eat eggs, cheese and other diary products. Vegans do not eat any animal products whatsoever. In any high tech company doing software these days, you can expect about 30% of the people to eat vegetarian-only and a small 1% or so to be Vegan. These are mostly the Indian engineers, and we need consider their needs as well. This means purchasing 100% durum wheat pasta (no eggs) or making a number of traditional rice and lentils dishes that do not contain butter, milk or eggs. And in some cases where stock is required in a sauce reduction, we need to use vegetable stock only. Since such dishes can be appealing to both meat eaters and vegetarians, it's important to make enough for everyone. That gets a bit tricky sometimes because some dishes may be labour intensive or costly and thus limited. For example, 2 years ago, at previous BBQ, I had a stack of portabello mushroom burger/steaks simmering in a marsala wine gravy that was enough for 30 or so vegetarians. But the mainstream folks loved them so much, it got depleted in less than 3 minutes and so quite a few vegetarians went without their main course. It still bothers me that I didn't prepare for that outcome and some folks went hungry. The obvious response by the omnivores was a bit callous - "It's BBQ Darwinism - Omnivores rule!"


This year, we chose to host this event at Sunnyvale Baylands Park - a municipal park that was both nearby and reservable for a fee. Another good thing was that this park had defined hours when they opened and closed, and charged all persons for parking. Such parks are usually fairly reasonable venues, costing just $300-$500 for site reservations, and discounted parking for groups on the order of $3/vehicle. By carpooling, the parking fees can be further reduced. We could have found a cheaper venue for sure that was only $50 to reserve and had free parking. But we chose the paid route this year for several reasons. The first is that the fees help to maintain the park in optimal condition. Free public parks tend to be more run down and lack clean facilities like bathrooms, electrical power and potable water. Second is that the defined hours and cost inhibit transients - especially the mentally unstable - from taking up residence in the park. Last year, when we hosted a previous BBQ in Milpitas, I ended up having to call Police when a young man in his early 20's on a BMX bicycle decided to crash the party. He began reaching down into his pants in front of a group of us, including our V.P. We've learned since then to be more selective about venues.

Heat Transfer

Make no mistake that in a BBQ, the key to cooking is mastering the Fire. The type of flame one targets really depends on the type and size of the BBQ. Typically, public parks have two types of BBQs. For single family and small picnics, they have a small metal box welded on top of a pedestal. It has a 1/2 sq. meter grilling surface and slots on the sides that allow the grill height to be adjusted. For larger group BBQs, many parks have masonry pits. These are often about 2 sq. meter area rectagular pits recessed inside a waste high masonry structure. A heavy wire mesh screen on a chain and pulley allow the entire grill to be raised or lowered over the pit via a long crank handle or cog wheel.

I know some folks who bypass public BBQ pits and lug their own grills because the public units are either broken or filthy. While this is sometimes necessary, often, a stiff wire grill brush plus a very hot fire are enough to rejuvenate even the most gnarly of public grills. Fortunately for us, the Pits at Sunnyvale Baylands are in premium condition. They are the big rectangular recessed pit types and each have potable water on one side of the grill and an electrical outlet with 110 VAC power on the other. Each reservable area sports 5 such pits and its own prep table, picnic tables, and garbage and recycling receptacles. Truly one of the finest public parks facilities I've had the privilege to cook at.

For public pit BBQs, I use a 3 stage fire that takes about 35 - 45 minutes to get ready. It starts with a modest amount of charcoal briquettes on the bottom. Any brand will do. I only use this as catalyst to pre-heat the pit. About 5 lbs shaped into a conical mound works for small pits. About 20 lbs (10kg) are more appropriate for the large pits. Note that the mound should be shifted over to one side of the grill. Which side? Well, take a sense of the wind direction and observe over a few minutes. Then position yourself on the side of the pit where statistically the wind is mostly at your back. This is critical when you cook so fumes from the smoke don't overwhelm you. For right handers, the main fire should be primarily on your right. Raw food starts on this side, and as it cooks, you flip it and move it to the left. Lefties should start fires on the left side. Don't worry if you don't cover the entire pit. You only need about 1 sq. meter of cooking surface per 75 people or so. You want some areas without direct heat underneath for food warming purposes but not cooking.

Next, squeeze a generous amount of charcoal lighter on the mound of briquettes. About 25cc/kg (about 0.4 fl. oz per pound of briquettes). Let it soak into the coals. Then light the fire. Wait about ten minutes and when the primary fuel has burned off, toss about twice equivalent weight of mesquite chips on top of the charcoal. If you like, you could switch and mix 50% mequite, and 50% hickory chips. You can buy BBQ chips from most BBQ supply places. Walmart has this near the garden section in 18lb bags. The key is to get blocks of chips that are about 5 cm in largest dimension but no smaller than 2.5 cm in smallest dimension. Technically these would be more like blocks than chips. Tossing the chips on the coals can snuff the flames out or reduce the flames initially. Don't worry, if the charcoal has been ignited properly as per the instructions above, it will reignite the chips in due time. If you're in a hurry and need to accelerate the process, get a piece of cardboard or a large, stiff paper plate and fan the coals for a few minutes to get oxygen into the center. This makes the coals red hot and when you stop, you should see a light blue flame shoot up as the adiabatic flame temperature really hits the 500 deg C mark the chips ignite readily.

Note that on rainy days where the humidity is in the high 90 - 100%, BBQs are very hard to light and start. Increase the fuel amount, and protect the coals and wood from getting damp. This is critical or else you make need to take much longer to fire up the coals. If rain is coming down, well, hopefully, you had a contigency plan to cater the food indoors and bake or broil the items in an oven. (But it doesn't rain often in October in California. We had 26 deg C weather most of that day. Just fantastic.)

Once the wood chips are burning well, which should be about 10 minutes later, take a fireplace poker or long sacrificial stick and spreadout the coals and chips more evenly. I then place oak or almond hardwood logs on top of the coals and let these catch fire. Again, about 2 or 3 times the amount of original charcoal by volume. I let this burn about 10 minutes more, until more than half the surface of the logs are white charred and glowing. It's now time to lower the grill above the fire. You should feel a tremendous amount of heat rising up as you get close, since the walls of the pit effectively insulate all heat loss from the side, all the heat is directed up. The heat may be unbearable. Usually, if I reach over quickly, just above the pit, the tips of the hairs on the back of my hands and forearms will burn. I highly recommend at this time to you have a small 2 - 5 gallon buck next to you, filled with cold fresh water for keeping your hands and arms moist. The evaporation of water will protect the skin up to 15 seconds as you reach over to manipulate food. But once your skin goes dry, you can receive 2nd degree burns easily roasting your hand and arm over the fire.

Note: For our event with 160 guests on the RSVP list, we expect about 15% more party crashers (because of the free food and proximity to our Santa Clara Campus). To accomodate such a large crowd, we decided on two pits to double the cooking surface. This also made sense since would could now toast a lot of vegetarian items on one grill with one set of utensils without fear of cross-contamination with meat and/or meat juices from the other grill.

Cooling down the drinks

Initially, such a fierce fire is not really suitable for cooking. Food would char instantly and the inside would still be raw. However, this heat is great for sterilizing and carbonizing any deposits on the grill. I usually let the heat do its job at this time and come back later with wet hands and a stiff wire brush to clean it before putting on my first items. Around this time, I prepare drinks, a thankful task because it gets me away from the pit and doing something that involves cold ice.

Lots of people think that with BBQs, the biggest task is the preparation and cooking of meat items. In fact, that's probably not true. If you deal with 18 - 29 year olds, the average male will eat only about 3/4 pound of meat (wt. before cooking) at any BBQ. Females eat just under 1/3 rd pound of meat. For the group 30 - 45 years of age with , reduce both down by 33%. That means for a group with average age of 32 years and 3/4ths males, and 1/4th females, we can expect 200 people to eat less 100 lbs of meat type items.

However, if the weather is warm, which it was 3 weeks ago - you can expect each male to consume close to 2.0 liters of liquid during a 5 hour period. Not all of it is actually swallowed, but people tend to waste about 25% of their beverages because they don't finish it before getting a new one, or they confuse it with someone else's beverage and then don't want to drink it. So on a hot day, we need to haul a lot of drinks. And the amount could even be more if most folks decide to participate in any physical activities like soccer (football), frisbee golf, etc. For 200 persons, this means having about 5 lbs of liquid each or about 1000 lbs of drinks. Try hauling that in a normal vehicle or minivan. It's quite difficult. And for most employees, such a monumental undertaking of hauling 1000 lbs of liquid is just not practical. That's where my wife and I come in. I drive a small Toyota Pickup. She drives a Sienna minivan. Between the two of us, we can haul about 1 tonne. This is sufficient for most warm weather BBQs upto 200 persons.

I still haven't gotten down to describing how to cool down the drinks. And this is a logistical trick. Basically, in warm weather, ice melts quickly. And it takes up volume so it's difficult to store. The key therefore, is to buy the Ice the morning you plan to use it. Or better yet, if someone on the team has an Uncle that owns the largest west coast Ice distribution company, even better. In fact, Kim, our expectant admin has an Uncle who has an Ice company, and we were able to get high quality (i.e. very cold and small chipped) ice in large quantities, and we could get it shipped to our site at exactly 11am prior to 12:30pm serving time. We needed a total of about 100 lbs of ice for the afternoon to chill about 1000 lbs of drinks.

With 3 bushel-sized large plastic drink tubs and two large capacity ice chests, we were able to load about 1/2 of all the drinks loosely into the tubs and put about 2/3rds of the ice into the ice chest. We then took ice bags and spread the ice over the drinks. The solid-to-solid contact of ice to beverages has very low rate of thermal diffusion and low Nusselt number. Such a tactic would take 2 hours to chill down drinks. To reduce time down to 15 minutes, a trick we heat transfer guys use is the 0.4 deg C triple point for water/ice/vapour mixtures. Basically, you can be assured that if you have a water/ice slurry, the temperature of the water will remain pretty close to 0.4 deg C as long as solid ice is still present. A beverage in contact over its entire surface with such a cool slurry will cool down to just 5 deg C (frige temp) in just 15 minutes or less. So the key is to add water to the tubs to immerse most of the drinks and then pile on the ice and somewhat fold it into the water to make a slurry. The Nusselt number goes way up as the liquid water not only increase surface area, but provides a fluid for much more rapid convective heat transfer over the ice alone.

Where's the Beef?

Good meat is hard to come by these days, at least at a price that can feed a hundred or more people for cheap. Thank goodness for Costco. The perfect store for these occasions. It sells a boneless beef chuck rib meat that comes in strips about a foot long and 2 inches in diameter. It's usually got a square cross section which is nice for grilling because it won't roll around, letting you keep track of cooking time per side. The meat is nicely marbled but somewhat tough along the grain with a few sinews. For just $3/lb, it's a bargin. It's great for stew meat and much more tender after a long cooking period. Plus it has great flavour being next to the rib. It is far more moist than a tri-tip or skirt steak at $4/lb, both of which get tough and dry. However, there's a reason why the meat is cheaper. It isn't shaped in anything close to that of a steak being long rather than flat and round. And it can be a bit tough just out of the package without treatment. Also, as far as steaks go, it isn't designed to be served as individual slabs. Instead, it needs to be cut across the grain into slices about 3/4 inch thick and served as small steakettes.

But, there is no doubt this meat can rival the flavour and satisfaction of a ribeye. The key is preparation. Most strips have one side that has a thick, attached membrane that was stuck to the ribs. This needs to be trimmed off and any excess fat removed. This comprises no more than 5% of the weight usually. Then the meat needs time to marinate and break down. There are two schools of marination - one that works and one that doesn't. One school recommends using enzymes to break down muscle fibres and protein, usually with some tropical fruit juice. But as steak aficionados already know, this destroys texture and makes the meat gritty.

The other way is to use some type of chemisty, usually with mild pH differences, and sharp changes in salinity or chemical concentration to drive cellular breakdown or dissication which is more akin to aging. This is exactly the same as brining meats, which makes them retain more moisture during the cooking process, but while in the brine, the tissues soften quickly. Depending on the kind of marinate and the degree of saltiness, temperature and other factors, you may only need to marinate for a few hours. But I like to go easier on the salt and marinate over night and kick in some other flavours. My wife discovered a brand of Korean BBQ sauce that isn't the standard tangy/spicy one used on Kalbee. Instead, this is similar to a Japanese style of steak marinade. Very low viscosity, but very dark. It seems to have some molasses and dark sugars, rice wine, salt, dark soy, and other ingredients. It isn't as salty or sweet or viscous as other sauces. This works very well on chicken and beef, and it especially works well with these rib steaks. A large zip-loc freezer bag with about 5 lbs of meat and about 1 cup of this sauce mixed with 1/2 cup of water seems to do the trick. Since the meat comes mostly defrosted but somewhat still solid in the core, a space saving trick is to use an ice chest and store the meat there overnight. The meat can melt and still stay as cool as the frige, and marinate more quickly.

Coq au Vin

Chicken and white wine just seem to get along extremely well when combined and cooked, releasing aromatic esters. My wife does an oil-n-spice mixture of light olive oil, Montreal Chicken Seasoning and some italian herbs and slathers this over boneless chicken thighs. Generous portions of chicken get shoved into zip-loc bags as well and before throwing them into the cooler, we pour a 1/4 cup of a good white wine, like a $2/bottle Charles Shaw Chardonnay you can get at Trader Joes. It takes less than an hour to process about 60 lbs of chicken this way, and the results are about a dozen bags of ready-to-grill poultry that takes about 15 minutes per batch.

Meanwhile... the BBQ Pits have settled down

After a good 45 minutes, the BBQ Pits settle down - no more big flames and smoke. It's very white and red and very hot. About 20 minutes before the bulk of the people arrive is when I usually take a final pass at cleaning the grill, and then start placing meat over the fire. The fire is not always uniform and there are always hotspots. In addition, the first 30 minutes or so, the flame may be so hot that you really need to watch the meat closely and flip it a lot. In fact, keeping the grill an extra 6 inches higher can allow you to keep up with the flipping. The chicken may have lots of oil residue and it can drip into the fire and cause flare ups. And the fatty parts of the beef can drip as well.

Keeping hands and arms wet and moist is critical early on. Also, having a long metal skewer, like a fireplace poker with a sharp hook on it is better for reaching across to the middle of the grill without sticking your arm over the pit and getting it roasted. Remember to use two sets of tongs and flippers. One for raw, and one for cooked. I make it almost a religious practice to only handle food with the raw food tongs until such time that the exterior is cooked. And then all the handling is using another set. This is the reason for having the fire on one side of the pit. The raw food is placed on my right side, then moves left as I flip it. As I take it off the rack on the left, I switch and use the cooked tongs. For most right handed folks, flipping by rotating the wrist counter-clockwise is easier and hence you want to start out on the right side and flip moving left. Chicken is done in about 15 minutes if it has been kept flat. If the meat is balled up, you will need to check the inside. Sometimes, it can take 25 minutes or more. So it's important to lay the pieces down flat originally on the grill. The steaks take about 20 minutes to cook because they are thicker. Individual chicken pieces can each represent their own serving. But the foot long strips of rib meat are better off being cut across the grain into 3/4 - 1 inch thick slices. This steak is still tender and juicy even medium well, so it is not necessary to have different batches to accomodate those that like their steaks rare or medium rare. Also, for such large parties, it's usually best to error on the side of well-done.

I don't recommend slicing the meat immediately. I prefer to have an aluminum tray or two trays (one for chicken, one for beef) at the colder end of the BBQ to put cooked items into. it's still warm for sure. But just not so hot as to continue cooking. Here, meats can stay warm, but rest for a few minutes. But sometimes, you have no choice if a queue forms waiting for fresh meat off the grill. However, if you did have time, ideally you'd let the strip steaks rest for about 5 minutes and then slice them. This pause gives the meat time to reabsorb some of the juices that were boiling inside and trying to move up through the meat driven by the intense heat.

The hot fire early on is great for cooking fast, and hopefully it satisfies the bulk of customers coming to feed. As the heat reduces, you can lower the grill closer to maintain the cooking capacity, or leave the grill at the same level and reduce your urgency by which you watch the food. Then maybe you can change shifts with an alternate grill-master and schmooze with others.

Sausages, Portabello Patties, Garlic Bread, etc.

Cooking sausages is the same as cooking other meats. The key is to use tongs and not a poker. We don't want to puncture the skin on a sausage and thus allow the juices to escape. Sausages are best left till after the fire is a bit cooler. This is because they tend to have high fat content and can catch fire easily when the grill is extremely hot. It's better to slow roast them. Sausages are ready when the skins get translucent and split or holes start to form and allow juices to escape under pressure. This means the center has reached a pressure above 1 atmosphere, and thus inside the sausage, with the salinity factors, the temperature must exceed the boiling point of water. It's a good practice to leave a sausage that is squirting on the grill for another minute to ensure that the interior is evenly cooked. Afterwards, I also like to slice sausages into bite sized chunks. Those who like it can take more. Those who don't, won't waste as much.

For vegetarian burgers, garlic bread, and other non-meat items, the grill will cause a major reduction in moisture. Garlic bread, because it will toast so quickly over the hot coals, this isn't a real problem. But with any type of vegetarian patty, they can turn into cardboard - literally - they will taste and chew like cardboard. What can work is to pre-fill an aluminum tray with a rich, vegetarian broth and red wine reduction with some type of tomato-base sauce. Off-the-shelf BBQ sauces that are thick and viscous can work here. Taking one part sauce, and one part stock and adding a teaspoon of red wine per liter of total liquid can make a very nice mixture. All this can be placed and mixed inside the aluminum tray, then placed directly on a hot part of the grill. The sauce will simmer after about 10 minutes and begin to reduce. As you finish grilling the Portabello patties, you can then dump them into this sauce and preserve the moisture and add a lot of flavour and also keep them hot. Folks who serve themselves will then have a very nice, moist and rich sauce taste on a vegetarian product. The only problem though is that the non-vegetarians may like this too, and deplete your resources quickly. I came prepared with 2 x 28 packs of vegetarian patties.


Clean up is always a problem. If everyone simply picked up garbage and/or recyclables around them and put stuff in its place, the rest of the things would take care of themselves. But lots of times, folks abandon their messes and others need to pick up after them. I usually travel with all my service ware, supplies, cutting boards, knives, kitchen supplies in large plastic tubs with lids. Dirty stuff all goes into a special tub and gets taken care of at home, and I keep a separate one for small leftover stacks of plates, cups, and other paper products to be used for the next event. Packing and gathering leftover stuff takes less than 15 minutes usually. But I try to allocate 30 minutes for final walk through to clean up the place, and another 30 minutes for sorting garbage from recyclables. It's amazing how many folks will be careless about throwing away a recyclable into a trash recepticle when the recycling bins are just 2 feet away. But allocating time for clean up ensures that next time, the park maintenance folks will welcome you back. Oh, having rolls of paper towels, a pack of food service plastic gloves and a jug of disenfecting wet wipes are pretty much de rigeur for us BBQ warriors!

Total bill for the event came to about $861. The total number of participants could have been about 175 or so. It's hard to say. It sure was a LOT easier hauling stuff back. I was left was just a few bags of chilled meats and supplies, and less than a dozen bottles of water. At $5/person, that's good morale boosting, I think.

Thursday Sep 16, 2004

Triple Boot Laptop, Finally

I've finally gotten off my lazy derriere and put together a Triple boot laptop. Surprisingly, the hardest part of the whole thing was installing the new, larger hard drive. My primary laptop is a cheap $750 Toshiba unit with 20 GB disk that I picked up about 3 years ago on sale with some big coupon, plus it had a $50 rebate. It was one of those non-expandible FRUs (Field Replaceable Units) with drives that couldn't be upgraded. But I purchased it at the time thinking otherwise.

Being a geek entitles me to feel overly confident about any piece of consumer-rated computer hardware, even though I may have no clue what the heck I'm getting myself into. And as Murphy's Law would have it, you bet I spent a good 3 hours the other day poking around the clam shell before giving up and calling Toshiba's Service Center to get a quote on how much it'd cost to install a new 40GB drive I'd supply. I got a pretty prompt reply: $89 labour. Or for $129, they'd transfer data to the new disk and extend warranty on the whole laptop for another year.

Uh... price was way too high. 'Okay, time to really put some brain cells to use and figure this problem out,' was what I thought to myself after getting the price quotes. As luck would have it, I did find out how to open the case, plus I didn't break anything doing it. The secret was to attack the screws holding the top LED panel cover over the LCD hinges. There are 4 screws in total, two 6 mm long and two 3mm long with mini phillips head. With a flat, thin prying blade and the LCD panel bent back until it was almost fully open 180 degrees, I could pry and pop of the LED cover plate. It revealed 6 more screws that anchor the keypad and top half of the clam shell to the bottom. I removed the keyboard as well, and then turn the laptop over and remove the dozen or more long 18mm screws around the edges of the clam shell. 4 more large screws anchor down the LCD panel, which I remove as well. Then there are 3 more in the mid-section that hold the clam shell together. With enough screws taken out, I could open up enough panels to gain access to the floppy and hard disk drive bays. With almost everything taken out, I decided to clean the built-in mouse track-pad and buttons. In total, I think it was over 48 screws, or at least felt that way; just keeping track of all pieces and screws was pretty hard. I definitely don't recommend you do this with young children around.

I actually got pretty lucky re-assembling the unit. The first time, I only had 2 screws left uninstalled. We all know that having extra screws pretty much sucks; you never know if it's some critical structural or disk drive screw. But I looked at it from a positive point of view. I could have forgotten way more screws. I took the laptop apart again, and this time found where the screws were supposed to go and filled 100% of the empty screw holes.... well I think I filled them all...unless the kids took some of them..

With the hard drive in, installation of software is pretty straight forward. I had to install WinXP first. The WinXP software is a recovery only type of install. It formats whatever disk is in the main drive and then creates one monster partition and sticks WinXP on it. The Win32 install makes a lot of assumptions that it owns the laptop, and thus, doesn't have to care about installing other OSes or playing nice. It will overwrite and hose everything else on the laptop.

But that's not a problem. The Open Source community has come through with a SysRescueCD image that contains a mini Gentoo Linux distro and nifty partitioning utilities that come on a bootable CD. Size is about 110MB for the iso image. Two included utilities, QtPartEd and NTFSresize are very helpful and low cost for resizing FAT-32 or NTFS partitions.

To make a triple boot system, I needed to first shrink the disk slice used by Win32/NTFS, then configure the remaining space into three slices, one for Solaris and the other two for Linux. Solaris needs to install before Linux for several reasons, one of which is that I want the Linux Grub boot loader to boot all three OSes.

I used QtPartEd to make these three additional partitions, okay technically I have to do 4 operations. First operation is to create a primary Solaris partition adjacent to the NTFS slice. I don't need to further sub-divide this slice for Solaris swap and Solaris UFS because Solaris will do it for me during the install and stay within its slice boundaries. The rest of the disk, I make a big extended partition. And inside the extended partition, I make two slices, large Linux ext3fs and small swap slice. Just the additional two big slices were enough. QtPartEd doesn't have a way to create and format Solaris UFS partitions in its menu. And after a first glance I wasn't sure what to do. But I recalled that Solaris x86 partition IDs are the same as Linux swap, 0x82. This can present a problem when installing Solaris and it sees Linux swap. It will try to use them and mount them as Solaris primary partitions, possibly installing on them. To avoid this possible snafu, we create an extended partition and put all the Linux partitions inside. The Solaris installer won't look inside the extended partition. So Linux swap inside the extended partition is safely hidden from Solaris.

Installing software was pretty straight forward. All the distributions came on CD, so the standard mode of sitting around and inserting the next disk are in order. WinXP recovery has 3 disks for my Toshiba. Solaris and Linux each have 4 total for the full distribution with documentation and multiple Locales. Installation time was about an hour for each. And WinXP and Linux each have over 500 MB in updates and additional software to download and install, such as service paks, updates, additional browsers, email, office utils, etc. Solaris 10 has yet to ship and so doesn't have a big list of updates, it may suffer from drivers or lack of them. Hopefully driver problems won't impact folks out there. The graphics and network drivers are often the culprits and the key is to bypass the graphical install and move on and fix later. I'm impressed because Solx86 has come a long way on x86 drivers in the last 3 months. This Friday Sep 17, Sun's Alan Duboff our Solaris x86 Technical Ambassador, will host another install-fest internally. I'm eager now to try the Solaris OS update features on the latest builds.

Some interesting observations about the other installations: WinXP Home took about 5 hours to fully install. The first thing it did on boot was to notify me of extremely urgent OS updates that were super critical to the health of the computer. To some, this is a great feature, but to me, it was kinda scary. I felt quite vulnerable during the first boot as I madly scrambled to download the patches and then startup the Network control panel up to block all further incoming connections from outside. I had to fork over money too. It's $50 to download Norton AntiVirus 2005 and get it installed. But I guess I didn't want to wait to head out and buy some OEM bulk copy for the 2004 version for $10. Definitely boot a fresh install of WinXP from behind a firewall/NAT router. Make sure it's the only Win32 machine running on the private net. Don't connect it to even a LAN that might have viruses active because if you haven't had a chance to reboot with network on to set the firewall one in XP, then you could be infected by some RPC virus right off the bat. But I guess that's the cost of doing business with Win32. In retrospect, the download of XP SP2 took so long, maybe I should have headed out to buy the OEM virus protection.

With Linux, it would have taken quite a while, except I archived updates on a server at home with lots of disk. There were 400 MB in RPMs and this includes new multimedia packages. I use YUM, the YellowDog Linux Update Manager, to download these updates. It's quite brain-dead easy. But the tip here is to run YUM once configured to download and preserve RPMs after installation (rather than blowing them away) and then archive the RPMs and install other machines with updates to save on bandwidth and time. That's why it pays to keep all desktop systems in a home or small business updated to the same revision of OS. By default, the Windows Updater obscures where it puts temporary packages that it stores for updates. Each machine therefore has to run its own update. I guess it helps eat more bandwidth, which may or may not be a good thing. On the flip-side, Win32 is so insecure, would I really trust a home machine to store archived updates? Not if my Dad was a user. He has all the spyware and virus blockers and anti-spam filters, yet he still gets about 1 nasty spyware per month and it's saved and runs out of the IE cache. I haven't upgraded him yet to Firefox because I never have the install media when I'm over at my parents' house and they use dialup so downloads are out of the question. Again, it's a huge cost of doing business, even for home users, on Win32. If I weren't around to provide tech-support, he'd be toast and so would his online stock portfolio.

Well, I've got my Triple boot system. It was pretty straight forward to do. Just a few small procedures to follow and I now have a small buffet of OSes. With a 40 GB drive it's possible to create a shared user-data partition in extended space and install quite a few OSes in smaller, 4 GB slices which then all access that shared slice for home directories. I've debated configuring a laptop this way. Sometimes less is more. Sometimes more costs too much. Sometimes paranoia sets in about the integrity of that data partition if the wrong OS were to boot and mount it. One thing is for certain. In a year, I'll revisit the whole decision again and probably re-install something else.

Sunday Aug 22, 2004

Getting more Nines on Wi-Fi Availability

Reliable, Available and Scalable - RAS - is almost like a mantra we mumble to ourselves in Enterprise computing. You've all heard the story of the renovated University building at Stanford or Berkeley where some Sun box running as a department mail server was walled up behind sheet rock accidentally, and it remained that way for 5 years and kept running until some decided to upgrade the system and couldn't find it anywhere, but they could still ping it.

But I had a little related "thank you" email come across from a neighbour up in British Columbia where my vacation home is. It's half way up Hwy 99 toward Squamish/Whistler next to a fancy golf course and next to the water. Well, about 1/3rd of the neighbours are residents from the States. Last Christmas, Telus (a.k.a. BCTel) finally dropped some fibre down from Hwy 99 (which they laid 18 months ago!) into our complex. We all got broadband at a clean 1.5Mbps down/640kbps up. I planned a Christmas/New Year's trip up at the time just to get the network up and install some wireless. That way, I could kick back out on the water fishing and still be logged into work. It always seems like the fish bite better when I'm not paying attention to the rod, so surfing the net just invites more hits.

I keep the Wi-Fi network open to the neighbours and put up two access points on opposite sides of my house. The units also sport high gain attennaes that push the signal clearly out to Hwy 99 which is 1/2 km away. So folks on the Golf Course should be able to get clear signal as well. Also, I set one AP to channel 4 and the other to channel 10 to support more users with fewer collisions.

My US neighbours just love the WiFi. Most head up there to ski and golf several times a year. And they've gotten used to the very reliable and available wireless and it saves them the monthly fees and hassles paying for their own connection, and insuring that it's up and running and secure when they arrive up in B.C. every couple of months. I simply donate the bandwidth. It's a small cost compared to what I pay for down here in the Bay Area for my DSL and the signal is so much cleaner up there too - as if I was just next door to the C.O. Plus with the exchange rate for $CDN, the price is even better.

So one neighbour wrote me a pleasant thank you email that expressed some amazement. On a recent trip, they had a power outage in the complex for about 30 minutes (a frequent event that happens once a month or so). But amazingly, they had laptops up and running, and the network never waivered. He said that he almost came over and knocked on my door because he swore I must be up there in the house working and maintaining that Wi-Fi connection because it's ALWAYS up. Even during the power outage. Whatever I was doing, Kudos.

I smiled when I read that. I guess what he didn't see when I came up during Christmas were the dual 50lb UPS backup power units I had. Each was connected to the AP plus the DSL router and switch. I chose the components carefully. Not so much for performance as I did for reliability and power consumption. I also have learned that less moving parts means more reliability. So I didn't put a running server up as the firewall/router, but used a solid-state off-the-shelf one that only has a limited number of ports. This way, I have battery backed power always available for the network and it's enough to power the entire network for 7 or 8 hours. Which exceeds 95% of outages.

So why back up the network? Because the Network IS the Computer. That's another mantra our company has preached for like that last two decades. But more importantly, I've learned from my mistakes. It was pretty embarrassing a few years ago when I was helping a friend setup his server for a Linux startup. The whole rig was in his garage. We bought a boat load of big UPS's to back up the servers. But we completely forgot about the network and on our first power outage, the servers were fine, but the network was down. That was pretty stupid and I've gone on to refine how I get more reliability into my networks. Some tips I remind myself with:

  1. With UPS, size matters. Bigger means more power longer.
  2. Cheapest and Simply are often better. Instead of forking over big bucks on name brand UPS power for your home, the biggest and simplest way to get power is to daisy chain a couple of deep cycle 12VA batteries to one of those plug in rechargeable Outdoors/Jumpstart camping power packs with CAR/12VDC cigarette lighter socket.
  3. To support multiple devices, buy a car cigarette lighter socket power-strip. It's got like 4 or 5 sockets. Then for each devices, I get a DC-to-DC converter that has a Zener diode to down-step the voltage. So this is why I select network devices for their DC power requirements. As long as they are under 12 VDC, I can buy/make an adapter cable that will power the device.
  4. Forget those 12VDC-to-110VAC inverters. Too much power loss in the conversion to AC and conversion back into 9VDC/1.5 amps. You get LONGER backup times going DC-to-DC only.
  5. Don't put a PC running Linux as a NAT firewall/router/DHCP server, even if it's a VIA Eden MoBo unit. It's less reliable and eats LOTS more power and requires a tonne of maintenance, plus you still need to power a switch. Most common WiFi AP routers have all these functions plus remote manageability.
  6. Wi-Fi APs with router and switch are much more integrated, lower power, and more reliable than separate switch, router and dumb AP.

Wednesday Aug 18, 2004

Eviction, quiet servers, and IPO


I got a notice last week from Workplace Re-location folks that my current office in Menlo Park is now slated for flex office space. In short, I've been evicted. But I didn't lose my office. I just moved around the corner to a bigger and brighter one.

Don't get me wrong. I think flex is a great idea that works for many folks. Coupled with Sun's iWork initiative where employees work remotely over VPN from home, it's really given us freedom to work where we need to be. In fact, I'm actually working in a flex office right now in San Jose because I needed to attend meetings here, today.

Sometimes I wish there was an option for me to go flex, too. But the WR folks haven't got an option for us engineers that have development systems that we need hands on with. But I have an idea how they could do it if any of them are reading this. Basically, in addition to flex-offices, WR folks need to provide a flex-racking space to permanently host our development systems and have terminal server console access and Lights-Out-Management on those boxes. And instead of an office to put these systems in, we are given locked drawers or lockers, and a shared office with a couple of desks and KVM-switched keyboard, video monitor, and mouse where we can attach to any of the systems in the rack. The only requirement would be of course that we still have physical access to our boxes on this flex rack, and that the flex-desk space is separate from the flex-rack so we can work in peace and quiet, and that we have adequate cooling, backup UPS power, and unfettered network access and shared workbench areas.

Some might call that a lab space. But labs tend to operate on a shorter-term project-by-project basis. They are large rooms filled with benches and noisy racks. And many have special network configurations, or physical access controls. And labs tend to be away from the mainstream office space and this reduces the local watering-hole/breakroom socialization aspect of office space in buildings.

Peace and Quiet

For now, I'm content with the new office. It gave me a new opportunity to clean out my existing junk and consolidate hardware plus reduce the noise level in my workspace. With 4 servers running in my office, the deciBel level can be deafening. I pretty much need to mute my phone all the time in conf calls. Power supplies are one contributor, but increasingly, I've noticed that disk drives have been getting really noisy. And this isn't from the newer, faster RPM drives, but from the old small drives; a sure indicator that bearings are shot on some of my older disk drives.

One box in particular was a dual-300MHz cpu Ultra-2. It had dual 4 GB SCA SCSI drives occupying the two bays, and these things were whining badly. No surprise since the box and disks were over 5 years old and average uptime between boots has been 200+ days! It had been serving as the group web server/java app server for quite a few years now and was low on disk and noisy. It's response was still snappy however, and it'd be a real waste to scrap the machine and pay for newer capital when really, it just needed a disk upgrade. Since I had hot backup on another machine already, I didn't need to replace the drives with mission critical hardware, so I sourced eBay and other local surplus stores in the Valley for SCA SCSI drives. To my surprise, I found a store in San Jose that carried 9.1 GB SCA drives, refurbed, from $2.99 and up. I picked up some 10K RPM Quantum Atlas II 9.1 GB Low-profile drives which formatted perfectly and worked great. And they were just $19 each with warranty. In fact, these drives worked as well as any newer SCA/LVD/SE SCSI drive. I just powered down the system, took out the old drive, removed the Sun spud bracket off the old disk, and put it on the new disk; then I slid the disk into the bay, booted the system, and immediately after the POST, I Stop-A the system and type 'reset' at the boot> prompt. I can confirm that the disk is found by Stop-A again after the reset operation, and running 'probe-scsi' at the boot> prompt.

Replacing the boot disk was pretty simple. I had upgraded the box about a year ago to Solaris 9 from whatever it was before that. I wanted to keep the boot configuration exactly as it was. So with the boot disk still installed, I powered off the system, installed the replacement drive in the second bay, reset and reboot the system, then formatted the new drive, partitioned and labeled it. Then I created a UFS filesystem on the disk for /root and any other partitions, and then mounted that under the current FS as /mnt or other mount point. Then I ran:

       # ufsdump 0cf - /dev/rdsk/c0t0d0s0 | (cd /mnt; ufsrestore xf -) 

Afterwards, I ran the installboot(1M)

       # installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0 

And then I powered down, and physically replaced the first disk with the second, and rebooted with reset. With a new second drive as well and the /export/home directories restored on that second drive, I now had doubled the disk capacity cheaply and quickly, I've now reduced the noise from the failing bearing to a dull buzz of the power-supply fan. Not bad for more peace and quiet. I'm now on the lookout for some quiet 36GB SCA Low-profile SCSI drives at a bargain. If they have them discounted in the 10 pack, I'm interested.

dotCOM IPOs All Over Again?

So what's up with Google's IPO? I heard on the news that it was repriced somehow and SEC had not replied back with a confirmed IPO date. I was amazed to hear that it was originally priced at $130 or so, and they downgraded the IPO price to $85? That's a huge hit. For a second there, I thought is was the dotCOM boom all over again.

I'm not sure if I'm going to buy any. I'm a conservative investor-type. I follow the Peter Lynch model of buying what I know and buying stocks of companies that I'd buy from. For example, it's not hard to invest in a sound enterprise computer hardware/software business. If the hardware and software solutions are compelling for a large enough enterprise market and the margins are acceptable, then the business model is purely based on execution and track record. That's a no-brainer for any investor. Then there are investments in consumer products companies. For example, I shop a lot on-line for outdoor gear. My favourite store is Cabela's. I've been a loyal customer for about 20 years now. BTW, they run ATG software on Sun, too. They have incredible customer service and great prices. They went public (stock symbol CAB) back in early July I think, and they've gone down a bit but come back a bit too. I buy what I know. And I know Cabela's gets at least $1000/yr in business from me.

With Google, I'm not sure. I've never paid for any of Google's services, so I'm pretty sure they make their money on advertising and through affiliations and selling subscriptions to their search engine software. But I'm not sure what kind of business it can be or what the volume is. If they started charging me to use it, I'd probably drop the service. If they got my telco to price it into the subscription model, maybe I might just pay, but maybe I'd switch to a new provider too. Or, maybe, I'd charge Google back for spidering my websites since my existence benefits them too... hmmm.

I know when a very popular online Free Email provider tried to charge me $30/yr for POP access, I cut my ties and migrated my Mom, Dad, Wife, and In-Laws to my server at home too. As the number of mailboxes I host increases, the move looks smarter and smarter. Plus, I'm getting better privacy since now I'm in control of my mail, and it ain't some lowly paid employee admin hack that can sit around and monitor my mail. And plus, I use the same open spam site filters that these guys use. Of course, I have to remember that not everyone knows how to host their own web/mail servers and that I do spend more in ISP costs. But even if I couldn't host my own email, My Dad and Inlaws currently use Access4Less.NET, a $5.95/month Nationwide no-frills and zero-support dialup ISP. POP mailboxes are included with that subscription. So there's no need to mess around with any of these freebie email providers that aren't really free.

So, if these free service dotCOMs aren't making money from me, it's an interesting question where the money's at. Maybe from a speculative position, lots of investment banks are simply waiting for secondary markets to buy up shares before they cash out and make off with a measely 100% profit. Small compared to investors during the dotCOM boom when investment banks and underwriters at even the second/third round were getting a hold of stocks and making 1000% in just a year. But I could be wrong and there is a business model in all this...




« July 2016