Thursday May 15, 2008

Sun Open Storage and the Sun Fire x4240

We announced a whole set of new servers this week, including one of my favorites, the Sun Fire X4240 Server. I only like to talk about things I've touched and played with so I figured I'd show you a little about how I'm using my x4240s.

I'm a storage guy, so what I immediately like about this particular system (and the Sun Fire x4150s I talked about before, as well as the related 8 and 16 disk offerings with SPARC chips), is the disk density I get along with the raw horsepower sitting right next to the disk drives.

The Sun Fire x4240s I'm using right now have 2 x Dual-Core AMD Opteron 2222s that run at 3.0 GHz, but you can have a couple of Quad-Cores in this thing as well! As for storage, I have a set with 146 GB, 10k RPM SAS Drives that are pumping about 66 MB/sec through a single NIC port with our Open Storage CIFS Server that integrates nicely with ZFS.

Watch me share 15 drives through CIFS so those poor Windows folks can attach to my box for their storage (and get the benefits of ZFS snapshot and capacity):

$ zpool create sharepool raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 raidz2 c1t6d0 c1t7d0 c1t8d0 c1t9d0 c1t10d0 raidz2 c1t11d0 c1t12d0 c1t13d0 c1t14d0 c1t15d0
$ zfs create sharepool/share1
$ zfs set sharesmb=on sharepool/share1
$ zfs set sharesmb=name=share1 sharepool/share1

(You also have to configure the PAM module appropriately, but this is a one time thing, try this:

1. Add the below line to the end of /etc/pam.conf:

other password required nowarn

2. Set/Change the Passwords for any Solaris User That Will be Used to Authenticate when Connecting to a CIFS share, like this:

$ passwd pmonday
New Password:
Re-enter new Password:
passwd: password successfully changed for pmonday

With the SMB PAM module installed, passwords are generated that can be used by Windows as well as Solaris (you have to do this). There are, of course, differences from your normal Samba installation but with everything, figure out the tool that is best for you and go with it ;-) One thing you notice with Sun's CIFS Server is there is no anonymous access to shares (this can be a good thing)!

Sun's CIFS Server and the Sun Fire X4240 are just one example of how the Open Storage initiative pays off. You can turn this capacity / performance rich server into an iSCSI target, NFS target, or even keep the storage to yourself and load applications directly on the box that can make use of the fast SAS spindles. Don't want to use Solaris or OpenSolaris...heck, load Red Hat or Windows on these things, we do it all the fact, we'll even pre-load a Linux-variant for you (check the options page).

Cool stuff, nice job System's Group! Oh yeah, and the iLOM in the x4240 works GREAT (better than the x4150 even).

Friday Apr 25, 2008

Paul [Hearts] eLOM

For the past week or so I've been on one of those "get this done" projects from my boss. Its tough to "get this done" when you are commuting though (or when the lab is the same temperature and wind speed as the top of Loveland Pass). My project involves 5 Sun Fire x4150s (and will involve another set of servers next week).

I did have to go into the office for a few days to pull memory in and out of them, do some DVD swapping on the machines, check cables, and (of course) get the 8 x 73GB 15K SAS drives swapped in (all of my machines need the same drives and speed).

During the setup, Terry Hill allocated IP addresses for the service processor eLOM (embedded Lights Out Manager) and configured the IP addresses within the BIOS. Keep in mind when you look at the back of a Sun system with eLOM, you see the standard ethernet ports (typically 4 on this size box), but you also see an additional port that's offset from the others and labeled "Service Processor". It is next to the Serial Console port. You can see it here, on the left side of the back of the system.

(The x4150 eLOM documentation has more details about configuring eLOM).

Once the network is configured, you type in the web address of your x4150 service processor ( and up pops the eLOM Web User Interface. You can do this from anywhere (even home :-)

Now that I'm in the eLOM, I can do all sorts of things. I primarily use the Web interface. With it I can do all of the power resets I need to do (including having it come straight up to BIOS). I can also launch a console through the magic of Java. When you press the button to get a console redirection, a tiny Java Web Start application comes down to your machine and connects back to the server. With it I can do all sorts of cool things like handle BIOS, take console options on my Operating Systems (I use Solaris, OpenSolaris, RHEL and Windows on my machines), and so on... I am also able to monitor event logs and ensure the components are operating as I expect (heck, I can even flash LEDs to say Hi to all of those frozen lab folks who haven't set up their eLOM).

My biggest complaint with eLOM isn't even with eLOM, its with Mac OS X. I have to tell you, I am not happy with their Java support at all. The eLOM console redirection doesn't work (strange Apple-only Java problem). I went to to get a Mac OSX updated luck, have to go to Apple. Go to Apple, they are hung up on Java 1.5. I found the Java Platform Standard Edition 6 Press Announcement, its been out since December 2006.

I found a Java Platform Standard Edition 6 Developer Release at Apple but it only runs on Leopard and I haven't upgraded (and I'm not). So, I'm stuck with the Mac...can't use it in our labs to support my systems. I know people love their Macs but some days it reminds me a little too much of OS/2. Great promise with a lack of application support.

I bounced to another platform (with its own Java 1.5 that actually works), the JVMs on my Solaris machines work fine as well as on my Windows laptop and home systems.

In the end...lovin' the eLOM!

Monday Apr 21, 2008


Have you ever been attacked by a plastic grocery bag on the way to work? One of those ones that float through traffic freaking you out? And then it seems to somehow manage to go under your car and pop out the other side only to terrorize yet another driver! Kai and I did some calculations with the bags.

Each bag is a bit over a 15 inch square.

If I didn't use my cloth bags and reuse the plastic "overspill" bags, I would use about 10 per week (why they pack a single bag of potato chips in another plastic bag all by itself is pretty far beyond my comprehension). So, I work my butt off to save at least 5 plastic bags a week (usually I can save all 10). Its probably reasonable that every family could cut about 5 bags off their plastic bag diet if they haven't already.

Saving my 5 bags each week saves about (5 bags \* 52 weeks) 260 bags per year. That's saving about (260 bags per year \* 15 inches) 3,900 bag inches long (15 bag inches wide) and that is about (3,900 bag inches / 12 inches per foot) 325 feet. Each year saves about the number of bags to go across an American football field and an end zone.

(Yes, that's the high school rugby team playing on the football field...I know they don't need a bunch of plastic grocery bags flying across the field...and they do, quite often).

Let's say all of the families in Highlands Ranch (about 20,000) stopped using about 5 bags a week too. That would be 325 feet \* 20,000 people = 6,500,000 feet (15" wide). That's 1,231 miles of bags (15" wide).

The folks of Highlands Ranch could save a strip of plastic bags that runs from Denver to San Francisco (as the Google Map flies).

View Larger Map

Well, that's a lot of bags. Who knows the countless number of car accidents that would be avoided if we all saved a few plastic bags. Reuse the ones you have and get, recycle them, and in so many cases...don't even use them in the first place.

While thinking about the surface area that these bags cover is convenient and somewhat astounding, also keep in mind

  • Energy is used to produce these bags
  • The bags are comprised of materials like plastics and oils
  • The bags take up space in landfills (not much...but it is space)
  • The bags won't decompose for 100s or 1000s of years

Next time just think: "Do I really need that bag or can I just carry my sack of potatoes?".

Monday Apr 14, 2008

Power Consumption of Hanging Clothes out to Dry

I had a section to do for Kai to become a Wolf in Cub Scouts...luckily, the pack has decided to go out and do some neighborhood cleanup. Still, I was interested in completing my fact finding mission anyway. My goal was to create a set of information on how "little things" add up to "big things" (remember "think globally, act locally").

There is a lot to choose from in my life, I figured I would take them one by one and see if a little "group think" can help me fix any logic problems. After the Jack Johnson Curious George Soundtrack song, I figured I would categorize into:

  • Reduce
  • Reuse
  • Recycle

And I would take a few simple tasks a child could complete with their parents:

  • Encourage your parents to hang out 2 loads of clothes a week
  • Stop using 5 plastic bags a week
  • Use refillable water bottles instead of purchasing bottles of water
  • Walk to school

I haven't dug up all of the facts yet, but I'm committing to it here on the blog!

First task: Encourage your parents to hang out 2 loads of clothes a week.

(Yes, that's one of the loads I hung out today)

I have an LG Tromm Dryer (courtesy of a good quarter or two here at Sun Microsystems). It runs at 240 Volts / 30 Amps, ouch. That's about 7200 Watts per hour. A load of laundry takes about an hour to dry. I've committed to hang out 2 loads per week of laundry, 14,400 Watts per week \* 52 weeks = 748,800 Watts saved per year.

I was browsing around the web trying to find the conversion and I found the Ask a Scientist Web Site that said 1gm of coal could power a 60w light bulb for 550 seconds (let's do some rounding and just say 9 minutes). Now we have to do some normalization.

A 60 Watt light bulb with 1 gm / 9 Minutes = 6.67 grams / hour.

My two loads of laundry actually use 14,400 Watts per week (that's two hours of power), for 1600 grams of coal per week \* 52 weeks = 83,200 grams which, for my fellow U.S. citizens is 182 pounds.

Another site estimated the creation of 1 Kilowatt for each pound of coal (roughly), which would bring us to about 748.8 pounds instead of 182. The first calculation was in an ideal world, the latter appears to not be in an ideal world.

Doesn't seem like very much does it. I'll use the 182 figure to be conservative. Each of those pounds of coal produces about 3.7 pounds of CO2 so on the conservative chart, I am stopping about 673 pounds of CO2 from going into the atmosphere.

That's quite a bit, but is it worth the trouble?

My little suburb has about 80,000 folks in it, let's say there are 20,000 homes and each one of those families could save 136 pounds of CO2 from going into the atmosphere. That would be about 13,460,000 pounds of CO2 that does not enter our atmosphere each year.

Is that interesting yet? Probably not in the context of all of the CO2 created in a year, but I think that is starting to get interesting.

For a child, what is 13,460,000 pounds?

I saw on Yahoo Answers that a Pontiac Vibe is approximately (and conveniently) 2,700 pounds.

(Image borrowed from Edmunds)

My little suburb of Highlands Ranch, if each family would hang 2 loads of wash out per week rather than dry them, would keep about 4,985 Pontiac Vibes from entering our atmosphere and hanging over us each year. Imagine what a country like the U.S. could do by hanging their clothes out to dry?

If you see any MORE logic errors, please let me know and I'll keep trap and fix them.

Note: I was off by a power of 10 (sloppy night time work) and my dryer was 240 volts. Michael Lyle sent the correction and also noted that I should use an ammeter to get an actual measurement. Stay tuned!

Thursday Mar 27, 2008

National Archives, PASIG, a little Vacation

My family and I took a brief vacation this weekend and made our way to Washington D.C. for a little R & R. We enjoyed 2 and a half days of sights, tours, history and we even squeezed in a little time for the pool. For those of you that have been to D.C. (or live there), you know that 2 and a half days only allowed us to scratch the surface of the United States cultural base that is alive as well as preserved in the city (and often within a few blocks of the National Mall).

There are so many thought provoking and emotional moments as you move around that after two and a half days I found myself almost completely wrung out. We saved the Congressional Gardens with the Vietnam Memorial, World War II Memorial, Lincoln Memorial, Korean Memorial, and the others for the last day. The artistry and the thought that went into these memorials is astounding and the emotions that they pull out of you put you into knots.

I won't list everything we did on the whole journey over the weekend. For my youngest son, going up the Washington Memorial (and our need to start standing in line for tickets at 6:30am) will probably be the most impacting moments. For Shaun, hopefully the Vietnam Memorial and the Pederson House. For me, who knows, the Bill of Rights, the Constitution, the Declaration of Independence, the Magna Carta...simply amazing, but overall I can't name a single moment that wasn't worth its weight in gold.

Professionally though, the National Archives had to be one of the most thought provoking of our stops.

Here is this large building, with all of these physical manifestations of our history on display and in vaults around the building. The Constitution, the Declaration of Independence, the Bill of Rights, the Emancipation Proclamation and more than I could ever list here. It was "Magna Carta Days" at the National Archives as well and one of the four remaining copies of the 1297 Magna Carta from King Edward I was on display.

Here are a few thoughts that went through my head:

  • With infinite, perfect copies of digital content, what makes a digital entity "unique" and "awe inspiring"?
  • How is our country going to preserve and revere a digital creation 710 years from now?
  • How do you know what digital creations are worth preserving since you can hit a button and destroy them so easily?

The first of these seems entirely out of place with storage technology, but when you stand in front of the Declaration of Independence it makes you wonder what digital content could actually have this impact on a person and how you would embody that digital content. The record companies are struggling with it as well. In addition to digital download content, the companies are trying out releases on USB thumb drives as well as larger packages and "Deluxe" sets. Books are going to struggle through the same revolution (as magazines already have) of being bundled as bits with little or no "branding" or "artistry" about the packaging. How does one recreate that sense of uniqueness when the content is merely a bunch of bits that gets flattened into 10 songs amongst 8,000 on an iPod? Really, what "value" do those songs and books have anymore when they can be passed around at will and are part of a great "torrent" of traffic into and out of our computers? Something really has to "stick" to remain on "top of our stereo" these days.

And as for the Declaration of Independence...what a clean and simple document. The document itself hung in windows and is incredibly faded and worn down. Only after time passed did our country seek to formally preserve it for posterity. Perhaps we caught it in time to save it from deteriorating any more. But one still has to ask, is it the single original document that retains the significance or is it the content that remains significant. If it is the content, we wouldn't store the original in a huge underground vault and protect it as well as Vice President Cheney, would we?

Having seen the original, I would have to argue that there is something incredibly unique about it, it actually holds more reverence (for lack of a better word) than one of the many copies of it. So how does one reproduce that "reverence" in a digital world?

If that is not enough to think about, we have to think about digital preservation. The Magna Carta of 1297 has withstood time for 710 years and is in wonderful shape. What digital storage technology today do we have that can withstand decay for that length of time (of course, one could argue that some rock etchings have withstood time for 1,000s of years). Let's put this in perspective, today's disk drives and SSDs are generally spec'd for 5 years. If I want to preserve my family's pictures for 710 years, I would have to ensure the data was migrated 142 times. Hmmm, I'm not sure if my kids and their kids and their kids are up for that.

It appears that CDs and DVDs may have a lifespan of around 50-200 years if you preserve them properly. That is getting pretty reasonable...of course, they haven't been around for 50 to 200 years so they are certainly not battle tested like carving on a good rock. The National Institute of Standards and Technology appears to be looking heavily into the longevity of optical recording media. DLT appears to have a shelf-life of around 30 years if preserved properly.

Let's say, hypothetically, that you solve the problem of the storage media (perhaps a self migrating technology in a box that guarantees infinite lifespan and that, itself, produces the new disks and technology to ensure fresh DVDs are always built). Now you have two additional challenges (at least):

  • Maintaining the integrity of the data (how do I ensure) that the data that is NOW on the DVD is the original data
  • Maintaining the ability for outsiders to inspect and recall the stored information

The first of these seems obvious, but is actually quite difficult. Checksums can be overcome with time (imagine compute power in 700 years!) and we can't guarantee that the keepers of the information will not have a vested interest in changing the contents of the information. We see governments attempting to re-write history all the time, don't we?

Let's take a simpler example of what happens when "a" byte disappears. Recall Neil Armstrong's famous quote: "One small step for man ... ". Well, after a lot of CPU cycles and speculation and conspiracy theories, it turns out that we now believe that Neil Armstrong said: "One small step for a man...". It is a fundamentally different statement (though there is no less historical impact). This data is only 40 years old, but consider the angst in trying to prove whether or not "a" was a part of the quote. What happens when a government deliberately alters, say, the digital equivalent of an "original" 2nd Bill of Rights written in 2030?

One more thought for the day, since I really do have to work and if you have made it this far, it is my duty to you to free you of my ramblings.

We know for a fact that the English (dare I say...United States dialect) language is evolving. Even after 200 years there are phrases and semantics and constructs in the Declaration of Independence that require quite a bit of research for the common US citizen. Take the following paragraph:

He is at this time transporting large Armies of foreign Mercenaries to compleat the works of death, desolation and tyranny, already begun with circumstances of Cruelty & perfidy scarcely paralleled in the most barbarous ages, and totally unworthy the Head of a civilized nation.

There is the obvious use of the word perfidy, a word that has since all but disappeared from common speech in the United States.

Looking deeper at the paragraph we see evolution in spelling (compleat). There is also a fascinating use of capitalization throughout the Declaration of Independence. The study and usage of capitalization alone could be worth the creation of long research papers.

What does this tell us? The content and meaning of a work lies often with the context and times in which the work was created. How does one retain this context, language, and ability to read the content over 700 years? This is not a small problem at all. There are entire cultures lost or in the process of being lost as the language and the context is lost, consider the United States own Anasazi culture as an example.

A computer dialect (protocol, standard, information model, etc...) are themselves subject to evolution and are even more fragile than spoken language itself. A change in a capitalization in an XML model may break the ability of pre-existing programs for reading and migrating information, resulting in lost information. Once you break a program from 200 years prior, how much expertise will still exist to maintain and fix that program?

Crazy things to think about. Personally, I believe we are in a fragile place in our history where we could lose decades of historical information as we transition between written works and digital works. As part of my night job I'm trying to get more involved in the Sun Preservation and Archiving Special Interest Group (PASIG) to learn more about what our customers are doing in this area. I'm also trying to reorganize my own home "infrastructure" to be more resilient for the long run to ensure that my family's history does not disappear with my computers.

There are significant challenges in the computer industry all over, but preservation of history is one that our children and our children's children will judge us with. USB thumb drives will come and go, but hopefully our generation's digital treasures will not go to the grave with us.

Monday Mar 03, 2008

Computational Photography and Storage

There is a great article on CNet's about computational photography, "Photo industry braces for another revolution". It is basically about Photography 2.0. The first wave of digital photography seeks to reproduce film-based photography as well as it can. Photography 2.0 advances hardware while taking advantage of higher processing power within the camera to take advantage of the new hardware, replace hardware functionality with software functionality or bring image detection and manipulation capabilities that are not possible in the hardware space.

There are a few developments worthy of note, and all of them involve bringing more CPU capabilities into the camera:

  • Panoramic photography - I enjoy these types of scenes (one shown below), though I don't think they are the future of photography at all
  • Depth of field and 3-D photography - There is an excellent example of this in the CNet article. Personally, depth of field is arguably one of the most difficult techniques to master since this is purely 4-dimensional using our current lenses (aperture size decrease increases time of exposure and more depth will be in focus, etc...)

There are many other ideas in the article...detecting smiles (an extension of this is closed or open eyes), better light detection, self-correcting for stabilization (this is done with high priced hardware today in Image Stabilized lenses), etc... Clearly a Photography 2.0 revolution is in the works.

Photography 2.0 is really the same trend we see in the storage business...Storage 2.0. There are simple changes in the industry, like the incredible increase in CPU driving software RAID into storage stacks again. A huge benefit with software RAID is the decrease in hardware costs that it drives. This is very similar to the Photography 2.0 concept of moving image stabilization out of the hardware (the lenses) and into the software.

Storage 2.0 also brings us projects like this one: Project Royal Jelly. Project Royal Jelly encompasses two important pieces, one is the implementation of a standard access model to fixed content information, a second is the insertion of execution code between the storage API and the spinning rust. The ability to "extend" a storage appliance (or device) via a standard API will allow us to leverage the proliferation of these inexpensive and high-powered CPUs. A common use-case for an execution environment embedded in a storage device would be an image repository or a video repository. Every image submitted goes through a series of conversions: different image formats, different image sizes (thumbnail, Small, Medium, Large), and often a series of color adjustments. Documents go through similar transformations: a PDF may have different formats created (HTML primarily), the document will be indexed, larger chunks will be extracted into a variety of metadata databases for quick views, etc...

These transformations can arguably be the responsibility of the storage operation, not the application operations, especially when the operations can be considered part of an archiving operation. While indexing and manipulation could be considered a higher tier, storage tiering and taking advantage of storage utilities could also benefit from a standard storage execution platform. Vendors could easily insert logic onto storage platforms to "move" data and evolve a storage platform in place rather than authoring applications that have to operate outside of the storage platform.

Just some Monday morning musings...have a great week.

Monday Feb 25, 2008

DVD Player = Trash Bin

My nearby Hollywood Video shuttered its doors this weekend. My MVP membership was automatically canceled by the company. As far as the personalities go, I enjoyed all of the folks there, it was nice to get real recommendations from real people. As I stood in line with my son to purchase a few stray DVDs I talked to my son that in 3-5 years he would never step into a video store again. I got a few stray looks in line. The more I told him about never being in a video store or a music store again, the more I realized what a foolish purchase those plastic cases and metallic disks in my hands were.

I convinced my son we didn't need the DVDs and I promised he could login to Amazon Unbox on the Tivo and rent a movie when we got home. We set the DVDs down, left the line, and I walked out of a video store with my son for the last time ever.

We went home, rented "Mr Bean's Holiday" at home (no, it wasn't my choice) and I'm boxing the DVD player and putting away whatever movies I do have.

And here is a message for Blu-Ray. I haven't looked at a Blu-Ray player, I'm not looking at a Blu-Ray player and I swear I will never have one in my house, period (unless I purchase a video game system with one but then I won't get movies for it anyway). Revel in your victory oh Blu-Ray backers, but it will be short lived, physical media for content is short-lived. I actually think Microsoft XBox is fine if they can get the video downloads working ASAP on their platform. These are pro-gamers, renting online will not be a problem and I doubt pro-gamers show off their DVD collection.

When I don't have my Tivo with me, I can download to my Media Center PC and plug it into the TV through the S-Video port, works great, portable media, very nice.

One less black box in my house is a good thing. Thanks for your time of service DVD player, I'll show you to the door.

Friday Feb 15, 2008

Why one bit matters.

Sometimes I wonder why I'm in the field of storage. Its not glamorous. Its JBODs, RAID arrays, HBAs, expanders, spinning rust, and all of those things wrapped into enclosures with lots of fans humming. My background is varied, I wrote a file system for my Master's, I worked on one of the biggest Java Business Frameworks ever (the SanFrancisco Project at IBM), and I've danced between the application and infrastructure space more than once.

I often think about my "ideal" job, I've even pondered it here on my blog...and take note, the new Jack Johnson CD is very good and I am ripping it to 8-track real soon now. Personally, I love the field of digital preservation, XAM is in the right direction, and long term digital archives are important to people-kind.

But still, this storage business, there is something to it.

I watched my friend get their eyes lasered to correct their vision this week. While I was watching, I was able to sit with one of the assistants and pepper her with questions, it is an astounding process. Basically, as I understand it, the Doctors use the scanners and computers to

  • map the surface of each eye
  • analyze the surface to understand why the vision is incorrect
  • create several corrective treatments
  • the doctor looks at the corrective treatments and adds their wisdom to make the right decision (a lot goes into this, like the health of the patient, the age, their profession, whatever...)
  • the doctor may tweak the map of places that need adjustments
  • the updated map is loaded into the "laser"
  • the patient comes in, gets prepped, the doctor aims the laser and sets the program loose
  • the "laser" jumps around the eye zapping away
  • the doctor reassembles the eye
  • the patient goes home

Coolness. But then the geek in me took over, I asked what I could about the machine, backup generators, power, moving the data, mapping the eye, etc... But my head kept thinking about the storage and computer software.

What if a bit is wrong? What if the bits are stored away but due to some battery backup cache being down, it doesn't really get stored and the out of date map is actually in place? What if one tiny point "ages" and becomes rust and there is no checksumming to see it "rotted"? These are people's eyes, you know? Would you want to be the storage vendor that supplied storage that messed up someone's eye because you didn't get the signal / noise ratio on the cabling right?

I've been thinking a lot about digital photography lately as well. While its not people's eyes, it is still an incredibly fragile process. In fact, many of the world's best photographers still do not use digital, and for very good reason. Even when you purchase photographs, you pay a premium price for pictures that have not gone through the digitization process.

Think about this, if a person takes a picture, the CCD (or whatever they are these days) takes the light and transfers it to a memory card. The memory card gets transferred to a laptop hard drive (in my case), a variety of backups are made and I move many of the pictures to SmugMug.

That's a lot of storage along the way. Now, let's say (God forbid), my house burns down. I get my pictures back from SmugMug and one of my pictures has a bit that rotted away.

Now, that is one tiny bit of imperfection to some people. To a professional, that picture is no longer an original. At that point, you have to decide to toss away your artistic integrity and photoshop the point to be like the ones near to it, or just toss the picture from your portfolio. Either way, the picture is never the same.

How would you like to be the one that sold the storage unit that allowed the bit to rot or be stored incorrectly, or archived incorrectly and destroyed that person's memory, that one perfect picture that was meant to be a keepsake forever.

Well, when you think about it, building storage units and management for those storage units is probably not as glamorous as owning the software or companies that specialize in photo archiving, or "lasering" people's eyes, or storing original recordings for artists, or archives of space travel. But those folks have to pick storage units from a company...and if you are the company they pick and you fulfill your moral responsibility to supply checksumming in your file systems, and well-tested storage that may occasionally be late to market to ensure that a memory is not lost or an eye doesn't get know, that's pretty rewarding.

Cheers to all of my co-workers at Sun who believe storage is more than a spinning drive or a paycheck.

Sunday Dec 09, 2007

Metadata, CAS and the future of Digital Photos

I have my new Adobe Photoshop Elements 6.0 up in the background and I am currently working away at tagging all of my pictures. Get this, I have 8,477 digital photos since I started taking them around 1997 or so with my Kodak DC120.

Here is a picture from the Kodak, I was trying to figure out how well the camera processed different color bands.

I have a few scanned photos from before that and, in fact, I went on a scanning binge this Summer with old pictures my Mom had sent me...of course, I have lost those already and have to re-scan them.

My newer Canon Rebel XTi is very cool. I can't say enough about it. Finally a camera that I feel replaced my 35mm (Canon Rebel as well). I have a few lenses for it and am still practicing like crazy. My friend bought me a kick-butt lens for it, 67mm front opening and a fixed f/2.8 aperture. Here is one of my favorite "technical" pictures so far (not to say I don't have a growing library of pictures that I love too...more on that in a minute).

Back when I used my Kodak, vendors hadn't realized the importance of a non-proprietary format. The Kodak recorded images in a ".mix" format. These days its hard to find software that recognizes those images, .mix has not stood the test of time. Generally, vendors have learned from their lessons. Oh, wait...more advanced cameras allow you to take pictures in a RAW image format, this is an unprocessed, uncompressed image that is basically like a digital negative. RAW formats generally suffer the same limitations as those early picture formats, you are often tied to the manufacturer's software to edit the RAW images, then convert them to something like a JPG.

Today was the first real sled day here in Colorado (at least in the Denver-metro area). I tromped out with my camera and kid in tow and snapped a few shots. When I say a few shots, I mean about 150. Here is one from the zoom lens.

Now, let's talk storage. I wouldn't even rate myself an "amateur photographer", I am more like a "every few weekend" photographer. Still, I have racked up 1,000s of pictures, and the flood-gates have opened with my new camera. I have about 40GB of photos today and the challenge of maintaining them is growing. Here are my current set of challenges:

  • Backup

  • Search and Retrieve

  • Picture Format Compatibility

  • Printing

There are more challenges, of taking a good picture...but let's stick to the above. Reading a book like Photographing the Landscape: The Art of Seeing by John Fielder makes you realize what an art a camera really is.

So, here is how I am tackling my challenges with some technology tips for storage and folks like me...please feel free to let me know if you have additional tips, I can always use pointers.

Simply put DVDs don't work. I just don't like fiddling around with everything. So here is my current home-grown process.

  • Download pictures to my laptop
  • Take a pass at processing the photos (initial edits, ratings, etc...)
  • Convert favorites from RAW to JPG (to share)
  • Copy all of the pictures to a second computer

Then, every couple of weeks I copy the photos again using the Areca Backup Software out on SourceForge.

Now, I haven't experimented with it yet, but Adobe Photoshop Elements 6.0 finally has an online backup facility built in. Trust me, we are all going here sooner or later. Its crazy having all of your memories on a hard drive. And, further, if you think you will have to leave your house in a hurry some day (just ask folks in San Diego about this), you better have them on your laptop or on one of the new external drives. Semi-professional and professional photographers even build their own storage networks with JBODs and systems as storage heads. The The DAM Book: Digital Asset Management for Photographers is a great reference if you want to see an outline of putting together your own system for archiving.

Search and Retrieve
Search and retrieve is becoming my biggest headache. Thankfully, I was reading The DAM Book... as I started encountering this headache. Basically, your memory will fail and you will start losing track of your most treasured photos.

Here is the key: metadata, metadata, metadata.

Adobe Photoshop Elements has been worth every penny I spent. I rate the pictures and add tags to all of my pictures that will help me locate them in the future. I use top-level classes of People, Places, Events, Sports and Other. The Places are cool as there is built-in geo-tagging so that all of your pictures for a particular place can be found on a map (as long as you are Internet connected).

The DAM Book had additional guidance for tagging that is very useful (like how to effectively use the \* ratings). Now rather than searching around folders that I think were around the date that I took a picture, I simply click on the tags, put in date limits and figure out the quality of picture I'm looking for, a nice spread of pictures is displayed.

The investment in tagging my old pictures has been heavy and I'm still not done, but metadata is the ONLY way to manage your pictures. In my opinion, the XAM initiative is going to be huge, it basically makes this type of metadata search and retrieve the "top-level" storage primitive. Today, my metadata is heavily tied to Adobe Photoshop and the application. With XAM, applications use standard APIs for querying objects rather than files. My prediction...XAM or something like it will be a part of every major operating system within 5 years, making the storage primitive that of a Content Addressable Storage (CAS) device. With CAS/XAM type APIs, applications should be able to share the metadata and content rather than having it tied to proprietary "sidecar" files. Further, a good XAM implementation should let you use the information as standard files if you are really tied to your file explorer.

Picture Format Compatibility
My Canon's RAW format (.cs2) is thankfully compatible with Adobe Photoshop Elements. So I can edit my RAW files directly from the application. There is only one set of filters I'm missing which would be nice to have, but I'll survive.

But here's the thing, there is an open format from Adobe that is extremely interesting, it is the Digital Negative (DNG) format. It is sort of like a TIFF file, but open and with much more information packed in. The files end up about double the size of your original RAW file, but for a storage archive, DNG files make a lot of sense. The contents are still "RAW" but have been moved to a standard format that can withstand the lifetime of a digital camera (RAW files have no guarantee to live past the life of a particular camera or brand, thus giving you a potential headache when you switch cameras). DNG is not ubiquitous yet, so often times you cannot read the DNG file from software (other than Adobe's).

Still, my $$$ are on DNG to be ubiquitous in the coming years. I have not converted all of my files yet, but the day is coming...once I have them all tagged ;-)

Finally, printing. Let me give you a tip. Use your photo printer sparingly, it is simply not worth it. When you add up the cost of the printer, ink, and the headaches and combine it with the limited options of your home photo printer...just forget it. I can get $0.10 prints from my local grocery store now. I literally upload them, tell them to print them, and I can go pick them up in an hour. Its amazing. The quality at the grocery store was "OK", not exceptional...but your local photography store allows you to do the same thing. I print at a photo store by the mall for some even better prints...they are about $0.20 a piece for a 4x6. Kodak's online site kicks butt too, those come in the mail in a couple of days and the quality is incredible.

But here's a tip I learned the other week. The image sensors on the Canon's (and other dSLRs) blow up at a different ratio than a pocket camera. The dSLRs have an 8x12 ratio with no crop whereas the pocket cameras have an 8x10 ratio. I have only found one site that does 8x12s, SmugMug. The pictures arrived after a couple of days professionally packaged and looking exceptional. The price was pretty good too...there is a fee to be a member though whereas Kodak is pretty much free if you order something once a year.

And that, my friends, is a blog post that is way too long but has been eating me up for a long time.


Monday Oct 29, 2007

Street Direction Savvy vs. Being a Good Driver

Whenever I'm doing architecture or talking to managers and such about projects, I always have a collection of metaphors in my head. I've always wanted to pull this particular metaphor out at a party but I've never had the right moment, so I decided to put a blog together about it since it is one of the architecture tools I carry around in my toolbox.

As with all metaphors, there is the danger you take it too far so...let me know if I've done that :-)

In practicing architecture and implementation and especially discussing project planning with people I encounter situations where a distinction is necessary between Street Direction Savvy (best thing I could come up with) vs. what I could probably term as simply Being a Good Driver. What's the difference you may ask?

In the real world, we spend a lot of time teaching our kids the mechanics of driving things: a bike, a skateboard, skis, and eventually a car (by the way, I believe the legal driving age should be 31). The parallel in programming is a person that has spent a lot of time learning the mechanics of a language or a specific part of a platform: Java 2 Enteprise Edition, Web Tier User Interfaces with AJAX, Database Engineering (by the way, I believe the legal age for use of Java 2 Enterprise Edition should also be 31...ITS A JOKE).

(Image linked from eHow)

One thing you quickly realize when you start to drive is that the technical ability to drive something does not make you an expert in getting from Point A to Point B. What's worse, the shortest path from Point A to Point B is almost always mired in complexities such as what you are driving (using a bike to get between points may yield an entirely different route as compared to a car) and what time of day it is (traffic often sends you onto side streets at particular times in the day). Further, what you know how to drive often forms your decisions on how to get between the points (knowing how to drive a bike and then learning how to drive a car yields some remarkably inefficient ways to get between points...often including driving through a neighbors yard (this is frowned upon when you have 4 wheels and 2,000 pounds added to your vehicle)).

As a result, you develop street savvy that goes with the context of what you are driving. Bringing this back to engineering, I like to think that architecture is all about street savvy and the better you know how the architecture is to be applied (how to drive) the deeper an architecture can go. Not only can I provide a map to the engineering teams, but I can also tell them specific platform decisions rather than just wave some boxes and requirements of those boxes at them.

Interestingly, some people never learn how to drive things but they have remarkable street savvy. Other times we are asked to have street savvy but we don't know how to drive what the other person is driving...I was asked by one of the Project Blackbox drivers how to get from Point A to Point B and I was like "are you driving your semi or your car".

This happens all the time as an architect. You end up applying the knowledge of something else that you learned how to drive that is similar. For example, an architect with experience in Swing and Designing User Interfaces may have a valid estimate for a team that is building an AJAX user interface. More importantly, I feel, is that the architect has to be sure to inform people of the differences in various tiers and platforms. Many times you are asked about an individual and whether they could fill a role on another team. People with driving savvy in one tier are often not useful in another tier if they haven't learned quite a bit about street savvy. Someone that has spent their entire life writing scripts in perl may not be helpful building an AJAX Web User Interface, yet it is a time honored tradition that management asks the question of whether the perl programmer can fill a role that needs immediate filling in the UI tier.

More than anything, I believe you are morally responsible for the information you give to people. So why not be transparent when you tell a person that your estimate to get from Point A to Point B is based on a map built for a bike rather than a map built for a car. Also, when asked about resources, feel free to ask questions about the person if you are not fully informed...if you've known someone only in the position as a database engineer, ask if they have experience in the user interface tier and what type of projects they've worked on. Ask if they understand the abstractness of languages and not just a language, this can be a big help...we all know that someone that understands a clutch and how to drive a particular manual transmission will probably be quicker on the uptake of another manual transmission since they get why they are pressing the mysterious third pedal down.

The metaphor works if you think about, enjoy...and let me know if there is a party around that I can use it at or if I can refine it a bit more.

Tuesday Oct 16, 2007

Rockies in the Series - Whoo hoo!

It was a LONG night last night at Game 4 of the National League Championship Series (NLCS). BUT, it was worth it, the Rockies swept the D-Backs to go into their first world series ever! We compiled a TON of pictures, here are a few:

First pitch!

Last out!

The end of game scrum!

The Rockies pulled this off with a payroll in the Bottom 1/3 of the league (along with the Arizona Diamondbacks and Cleveland Indians). Only the 1-game behind Boston Red Sox are in the top of the league in terms of spending.

Pretty amazing to think the Rockies were one strike from wrapping up the season a few weeks ago.

I guess the lessons are

  • Money does not equal success
  • It takes a team to succeed
  • The game is not over until the last out
  • The season is not over until you are at home watching the World Series

Yes, the photos are a bit blurry...we were above the purple seat line (indicating 1 mile elevation mark), so yeah...could have been better seats but it was worth it!

Go Rockies!

Thursday Oct 11, 2007

Sha - XAM (Fall SNW)

My team delivered an end to end prototype of the Version 0.65 XAM API (XAM is the eXtensible Access Method) to the SNIA XAM SDK TWG today for use at Storage Networking World in Dallas next week.

Its pretty exciting, our Sun StorageTek 5800 Storage System already has a client API available from the Sun Download Center and is in the process of moving to OpenSolaris as a first-class project.

XAM helps to standardize the architecture of the API as well as specific language bindings for the API. With the architecture and API specifications at 0.65 to 0.69, there is obviously going to be changes to the architecture and APIs as it moves closer to release. Still, it is a great start with obvious promise.

The API itself is worked on by a broad range of companies that provide storage solutions (EMC, HP, not to mention Sun Microsystems). Application authors are also a part of the TWG membership to ensure there is an end to end view of the architecture. The XAM Architecture itself provides a top-end API for application vendors as well as a bottom end API for participation by storage vendors. Within the architecture are query capabilities and standardized properties that are a part content addressable storage systems.

I know this part may bore a bunch of folks, but here is an example of submitting a query that returns a list of all XSet objects (objects that contain metadata and data) residing on a storage system.

String sQueryCommand = "select .xset.xuid";
XSet qXSet = sys.createXSet(XSet.MODE_UNRESTRICTED);
qXSet.createProperty(XSet.XAM_JOB_COMMAND, true, XSet.XAM_JOB_QUERY);
XStream vXStream = qXSet.createXStream(XSet.XAM_JOB_QUERY_COMMAND, true, "text/plain");
System.out.println("Closed query stream");
XUID queryXuid = qXSet.commit();

The results get stored in the qXSet and can be read. The application can then go back and talk to each separate XSet for more information about it or to grab content from the object.

So, as you're milling around the demo area next week at SNW, look for one of these:

and be sure to drop in on Mark Carlson's XAM talk on Monday from 10:15 to 11am.

Tuesday Sep 25, 2007

Taking a few mins...

Sometimes you have to take a few minutes and chalk up one of those things you have always wanted to do. The interesting thing about digital cameras (combined with Google) is that they give you the ability to "compress" the learning cycle. Something that would have taken a person months and a ton of money to learn about now takes a week and a good memory card (oh, and lots of hard drive space). Of course, it is probably not as gratifying which gives our society the ability to catch up with our predecessors but continue to build...standing on the shoulder's of giants.

With that, here is a shot of tonight's full moon (I missed the picture of the werewolf in my backyard...sorry).

The shot was done with a Canon Rebel XTi with the following:

  • F5.6 Aperture
  • 1/250sec Shutter Speed
  • ISO 100
  • Focal Length 300mm

Enjoy :-)

Wednesday Sep 12, 2007

Pneumothorax Recovery - 2 Month Recovery Diary

Just to recap: 4 Spontaneous Pneumothoraxes in under a month (frequent enough that I called them "not so spontaneous pneumothoraxes"), 2 thorocoscopies (surgeries that go in and staple your lung then permanently "glue" it to your chest cavity), 6 chest tubes, 4 hospital stays, and 2 sessions of "aspirating the air and liquid" under a CAT scan machine. Its been about 2 months since my last surgery and try as I might, I haven't found ANY decent descriptions of recovery periods on the web.

So, I'm going to deviate from my normal "blog tenor" to document some of my own recovery. To start with, though, a few notes:

  • Everyone is different, listen to your Doctor and your body
  • My case appears to be much worse than the normal pneumothorax, so what I say may be worse than what you are going through
  • I fully cannot understand why someone would choose to wreck their lungs with smoking or any other lung-destroying activity...

So, two months along, I just came from a checkup with Dr. Hofer. I still have 200cc's of fluid (approximately) in the bottom part of my chest cavity. The fluid comes from the irritation and scarring of the lung to the wall of the chest cavity when they put in the sterile talc...gross huh? I've been very, very careful over the last couple of months...I have 8 obvious scars along my side from the procedures and innumerable other holes from needles from the aspiration sessions. My first 3 weeks after the second surgery were completely miserable. Lots of fevers, very uncomfortable laying down, and a favorite chair to lay in. As of right now, the fevers are long gone and I got myself off pain medication as quickly as I could and onto Advil. After about 3 weeks I was off the Advil completely and by now I've been Advil-free for quite some time. Discomfort still reigns, I've switched from sleeping on my back prior to the surgeries to sleeping face down...the weeks after the surgery I slept on my back but propped up on pillows. If I do one of those "half runs" where you are late to school to pick up the kids so you sprint out, I have to stop in about a block cause it feels like someone is squeezing the bottom part of my lung, yech.

Around a month I went out for a bike ride, 7 miles total. I'm not a biker at all so it was a change of pace. I went on a road bike but switched to my mountain bike for the next time out. I am usually a jogger/runner, but the biking was a lower impact and the mountain bike even lower. My first few bike sessions I came home with a "stitch" in my side and I had to do some serious sleeping. In all, I've done about 4 7 mile sessions, a 20 mile session and a couple of 12 mile sessions. The bike is where its at for athletic recovery in the first few months...I'll tell you that.

I have seriously thought about giving up running altogether, but for me it is a nice release. Further, to do an apples to apples comparison on my recovery, I figured I had better do some running. Dr. Hofer gave me the thumbs up for running today so I was off to the races. To give you some idea of where I'm at, prior to the first thorocoscopy, my "inspired volume" for my lung capacity was around 4,250 mLs. Now I'm around 3,000 peaking at 3,250. I am also seriously out of shape. My weight also dropped precipitously to below 135, I was 142 before the surgeries and now have a goal to push my weight to a more healthy 155 or so. For those of you who do not know this, skinny does not equal healthy and skinny does not equal in shape. My goals for my first run were pretty simple:
- Jog over a block
- Keep my heart rate down in my normal running range (approximately 155) and go as slow as I have to for maintaining it
- Stop if I hurt at all
- Don't push it

To be honest, the recovery is going to be as much mental as physical. I've turned into a complete lung hypochondriac...

It turns out I did FAR better than I had hoped:
- I jogged about 5k (a little more). I promised I wouldn't time myself but to give you a ballpark, my Bolder Boulder time (my personal best) was 10k in 42 minutes...I ran my 5k today in about 35 or so not fast, but steady
- My heart rate spiked towards the end to 172...ouch, I pulled up at that point and walked. The recovery period was very long too. A high heart rate probably reflects both my lower lung capacity and my being so out of shape
- I didn't hurt. After the first block or so that eery feeling of someone being inside your chest squeezing part of your lung was basically gone...for the first time in months! It did feel creepy on my right side still, but the glued up lung is supposed to not be felt at all. Dr. Hofer mentioned the nerves are pretty messed up on my right side so these feelings are to be expected. In some cases for my rather long hospitalization, he is saying about a year to be completely back to normal "nerve" wise.
- I didn't push it, 5k was good and not too much

And that, my Spontaneous Pneumothorax buddies, is the 2 month recovery diary. There are more people than you think who have been through this, so don't be shy about talking to people. As far as I can tell, everyone has recovered fully, even with the glued lung!!!!! Feel free to email me if you want to know some more about what I'm going through to help yourself, or if you want to sponsor my recovery (I'M JOKING).

Wi-Fi Wiki Space

All right, I'm over my frustration from yesterday's Starbucks incident. I realized that the search for free Wi-Fi hotspots is always full of adventure. Tables are always full at the good Wi-Fi spots and in other places the Wi-Fi seems crippled or simply well out of range due to the size of the location.

I'm a big fan of Free as in why not create a place where folks can participate in creating an up to date listing of hot spots? I know even some towns are one big hot spot (I believe Fort Collins in Colorado is one of these, but I'm not going to plug it in until I've tested it in a few weeks).

As such, I've created the Free Wi-Fi Wiki. This can help some of us frequent travelers or work-from-home people get out and participate in the world and embrace those that are bringing the network to the people :-)

I know, you've found a "secret" free hotspot and you don't want everyone to know about it since you have your own comfy chair and you don't want someone to spill coffee on it or break the springs. Well, you don't "have" to give it up. BUT, I would ask that if you do use the Wi-Fi Wiki ever, to find a free hotspot to plug in...even if you keep your favorite a secret ;-)




« August 2016