Saturday Apr 11, 2009

iSER integrated!

iSCSI Extensions for RDMA (iser) was integrated into build 112 of Open Solaris. The team worked their butts off, and, at the end of a very long and very hard six months (probably too hard) put back the project. This action culminates some 18 months of work for Peter and his team. iSER is a key aspect in the Open Storage platform for IB connectivity solutions.

COMSTAR now supports Fibre Channel, iSCSI, IB (via iSER) and FCoE. We have achieved a true multiprotocol SCSI target. Now it is time to productize!

Well done team!

Thursday Mar 05, 2009

COMSTAR update

Just a little update on some of our latest developments:

1. We are getting ready to release a new SBD lun provider that will support PGRs, some cleaned up management interfaces and a couple of bug fixes. Should be out in the next 5 builds or so. Anyone that wants to help us with testing, send me a private note.

2. Tim Szeto, a lead engineer of mine is spending time thinking (deeply) about things like dtrace probes for COMSTAR, MDB walkers for COMSTAR, and kstats for COMSTAR. Look for open solaris discussions, ARC docs (join PSARC for that) and a storied discussion. Your observations and requirements are invited. Hit me.

3. iSER is just about ready to integrate. Would have put it back in 110, but, quiet build and all that. Don't want to destabalize the mother ship.

4. SRP target is cooking right along. Check it out, sign up for the developer's mailing list and poke at things. ( Peter C is rocking and rolling on the community update stuff. Shout at him if you need something!

5. FCOE is so very close. Roger's team is cooking this one pretty hard. Check it out at

6. iSCSI for COMSTAR screams! We knew it would, and it continues to get great reviews. It isn't perfect yet, but, it is turning heads. Check it out!

Summary? COMSTAR coupled with the deep traceability of DTrace and the power of The World Class filesystem is changing the storage game. My team continues to kick butt. Together we are crafting a superlative open storage foundation. Hop on board the Open Storage train.

Monday Nov 10, 2008

Disaster Recovery in Solaris

Mike La Spina, an engineer in Canada has been a member of my virtual team for about 6 months now. Mike is a great representative of the Open Solaris community, a strong advocate for quality (he doesn't really let me get away with anything) and a voice of reason when it comes to VMware, iSCSI and of course Thumpers. He connected with me in an effort to get me to fix a bug. We fixed it, he continues to suggest ways for us together, and that, to me at least, means team work.

Recently Mike contacted me about his DR plan, and I asked him to make sure he pointed me towards the blog when he published. Mike just posted a day or two ago and it is a great read. I would encourage you to take a look at it. Mike continues to drive as a tireless ambassador for Solaris and iSCSI and ZFS. All cool stuff.

Provisioning Disaster Recovery with ZFS, iSCSI and VMware

Sunday Nov 09, 2008

Target cheat sheets...

Last week I presented COMSTAR futures to the DMA (Data Management Ambassadors). We talked about the strength of the COMSTAR framework and a fair bit about the adoption curve. The truth is, hype and pride aside, COMSTAR is a game changer for us. The naive implementation is merely third tier storage or interesting scratch disks. However, as people become more comfortable with the employment of COMSTAR, they are recognizing that the strength of COMSTAR isn't so much that you CAN do these types of things but rather you can essentially deploy world class data services at a fraction of the cost. And, it is easy to do. So the possibility is merely exciting. The savings is revolutionary. 

I also promised to post the matrix of targets... Here it is...

Component S10 Nevada Open Solaris 2008.11 Open Solaris 2009.04
Use Build 100




No, but you can apply on top.

Wednesday Aug 06, 2008

Getting there

14:46:53.401 Starting RD=run1; I/O rate: (vdblite); Elapsed: 60 seconds.
For loops: xfersize=524288 threads=4

 interval                            i/o         MB/sec   bytes    read       resp    resp       resp       cpu%      cpu%
                                        rate       1024\*\*2     i/o     pct         time    max       stddev   sys+usr    sys
14:47:05.071         1    1861.87   930.94  524288 100.00    8.589  22.702    1.302    44.0         43.6
14:47:14.030         2    1863.34   931.67  524288 100.00    8.586  36.584    1.272    44.6         44.0
14:47:24.026         3    1863.05   931.53  524288 100.00    8.585  35.810    1.192    43.9         43.3
14:47:34.026         4    1862.25   931.12  524288 100.00    8.586  25.890    1.293    44.3         43.8
14:47:44.027         5    1864.35   932.18  524288 100.00    8.579  28.986    1.126    44.0         43.6
14:47:54.024         6    1858.23   929.12  524288 100.00    8.607  65.024    1.710    44.6         43.0

Wednesday Jul 02, 2008

Multipathing for Tape

It was a dark and stormy night, the back up windows opened and shut with the regularity of a 90 year old pensioner's clacking of teeth as he snored with the flem filled nervousness of old age and incontinence. The data flowed from the backing store to fixed content archives like a humpback whale strains plankton. The administrator anxiously clacked her gum back and fourth against the two teeth she had recently had crowned with a couple of diamonds (one shaped like an "i" the other clearly an "o"...some of the geeks at work thought it was a one and a zero, but, they don't understand IO) as she watched the progress of her back up. All of a sudden there was a rigid silence as the status window showed zero throughput and the tortured silent scream of failed IOs bounced off the ear buds of the administrator. With manic obsessiveness the administrator chanted her mantra "If an IO fails to complete, is it ever an IO?" and she anxiously counted off the seconds before the back up timed out. ....Over and over again she chanted her mantra as the digital clocked ticked with the slowness and regularity of a clock:

7..8..9..."...fails to complete..." 17...18....19... "is it ever an IO?"....28...29... (occasionally she changed her mantra to "I think I can I think I can" as she took power hits of her RedBull) 47...48...49... And then, with the almost ecstatic relief one feels as a particularly large boil is lanced, the IO kicked back in down the alternate path and the back up proceeded. 

So, with a rather shameless head nod towards the "The Bulwer-Lytton Fiction Contest" I announce that we have integrated multipathing for tape into Solaris build 93. This is the culmination of a lot of work in both the ST driver as well as MPxIO. In particular, we have solved the problem of logical block addressing and error recovery as well as true multi-pathing to multi ported tape drives. Additionally, this is a platform for innovation as well. Now that we have highly available tape back up windows that are natively part of the Solaris operating system (you could do some of this before with FCP2 Error recovery, but it wouldn't help with a bad HBA or cable) we can look at bus saturation so we can attempt to drive down back up times.

A strong head nod towards our experts in tape and MPxIO. Well done Randy and Wayne

Friday Jun 13, 2008

Rock star guitar hero

Sumit is still a rock star. Here is his latest guitar riff. Seriously, ZFS, DTRACE and COMSTAR: What more could you want on a phenomenal Friday in June?

Thursday Jun 12, 2008

"New Sun code transforms x86 servers into storage arrays"

The guys over at Computerworld wrote a little blurb about COMSTAR:

Open-source Comstar software enables servers to act as block-storage devices

By Brian Fonseca

Anyone out in the world want to share a story, complaint, request or <ahem> praise? Shoot me a note.

For those of you living under a rock, check it all out right here at:

Wednesday Jun 11, 2008


The COMSTAR press is starting to roll out and it is very cool. My team has worked on COMSTAR for more than a year, and, people from Roger Dong's team and of course our test resources and our docs folks have spent a lot of energy making the project and making it successful. Lynn blogs about it here. I expect a fair bit of press over the next few weeks. Deirdre' helped us get the campaign rolling and it is all staring to come together!  My very first Chinese blog was about COMSTAR.

The Leadville Stack

I was reading a bit about some of the open source activities out in the wild and I stumbled across this blog post: 

It was very cool to see someone writing a post about something we worked for many years on. Thanks Matty!

I need a code review


I need a code review. Randy, one of my whiz developers is ready to integrate his MPxIO for tape changes in build 93. This is a cool project and pretty bleeding edge. Great work and we need a code review. C'mon... Give a guy some help!

Best code reviewer gets a gift package from me. Cool Solaris stuff. I decide who the best code reviewer is. My decision is final and can not be swayed unless the review is delivered on the hood of a brand new car. (OK, minivan, I have lots of kids now..)

Seriously, help a guy out. Heck, post the review in my comments section... Just make sure you include your email address...


Sunday May 04, 2008

COMSTAR integrates

Sumit and John and the rest of the iTeam integrated into build 90 yesterday. Wow. The culmination of nearly a years worth of effort and a foundation to the next generation of storage software. Well done guys!

Tuesday Mar 25, 2008


A couple of days ago I received a SCA (Solaris Contributers Agreement) from the guys at Storage Switch. Turns out they want to help us write source for our flagship ST5800 or Honeycomb product. They have a few cool ideas about how we can leap into the future and they want to do it with us in the Open Source space. I could wax eloquent about partnerships and symbiotic relationships and moving the value chain, but, quite frankly, if you read my blog, you have heard it before. How about a different take on the same value statement?

Open source drives value and innovation for our customers. By working closely with people that leverage our value chain, we make it easier for customers to deploy our products. We increase visibility by having talented folks work with each other and turning out cool products. This enables the entire ecosystem, and, at the end of the day, if the customer benefits, because of tightly coupled development teams, we all win.

 Welcome Clark and Andy! We look forward to a profoundly beneficial relationship!


Sunday Mar 23, 2008

Making stuff

I spent this weekend fixing my John Deere 317, a 30 year old lawn tractor.


It is a bit cranky, but ultimately it goes forward and backward, pushes dirt and snow around, and cuts my grass. It just works (sometimes). I spend a bit of time doing this and that to it, and frankly I have spent more time fixing it then using it. Part of me is frustrated because I like things that just work. Turn it on and it goes. It cuts, pushes, moves, etc. And, when the green machine is working, it does it great. The other part of me, that part that loves working with my hands, solving problems and making things work, enjoys fixing the green machine. (I don't call it the green machine because of the color but rather all the "green" I have to put into it. For those not familiar with American slang, green is another term for money.)

There is something satisfying about a tune up, replacing an accelerator cable, repairing the cutting deck, fixing the brakes, hydraulic hoses and ignition switch, as well as every single tire on the green machine. You might gather that I have done all of that over the last two years. It runs better now, and it gets a bit better every day. All that being said, the allure of walking into the local John Deere dealership and driving out with a brand new machine kind of gets me in that part that is uniquely male. You know, that part of most men that drags knuckles, scratches where it itches and generally thinks that dirt and shooting things is pretty cool. Yes, it it true that my wife doesn't understand the value of a tractor that has a cup holder, can tow a boat (we don't have a boat), dig post holes (no fence posts) or frankly any of that cool stuff. She just shrugs and chalks it up to one of those things she can't understand and doesn't really need to. Anyway, back to my tale. So the tractor I want costs about $12K. At this point I stuff that male part of my brain back into the kennel that it belongs in and I fix the green machine again. There is a trade off between really really cool, and more functionality and features then you could ever use in your life time. Sure, I want to be able to hook up the dozer blade to the tractor, but, what exactly would I doze? (If my wife reads this, ignore the rational part of this post...Danny needs a dozer...)

This is analogous to the storage industry. Sometimes you really really need a dozer that costs $12K. Pushing dirt around is important, and, if you have to mow a golf course, well, you don't want a pair of hedge shears for that. I am a huge believer in getting the tools you need to do your job. (My engineers run on some of the best stuff Sun makes for development systems).  However, other times you just need acres of cheap and deep storage. You need a box that does a few things very very well, and, doesn't cost a fortune. It puts bits on dense storage, costs you exactly enough to meet your needs, but no more, and it just works. Unlike my tractor, you don't need to fix it every ten minutes.

We are building the right level of storage arrays. We offer the high end, wicked fast, storage devices. We also offer starter kits, unique ways of viewing the convergence of compute and storage and well.... Hold on to your hats ladies and gentlemen, now we start some really interesting stuff... This is going to be a great summer, and one that we will look back on in two years and say "There.. right there is when we made life more interesting for storage at Sun..."

 Now, I have a slow hydraulic leak that I need to go clean up.

Saturday Mar 15, 2008

COMSTAR tradeshow

Thursday my team got a chance to show their work to an army of systems folks in Santa Clara. I really wanted to be there, but since I have another project about ready to deliver, I can't travel right now. Well I could, but the team doing the actual work would find ways to make my life miserable. So, like a good manager I stay home and make sure the team has the appropriate resources and coaching. Hopefully we will deliver on time.

So anyway, we showed the world (at least inside Sun) some of the new technologies that we are working on. The cool thing is that I can talk about it here. I can talk about it here because we are doing it in the open and we have a number of folks participating. Here are some of the essential components associated with world class engineering (Food, Excellent engineers and racks of wicked computers).



Qlogic, our first partner on COMSTAR, put together a poster to show the value chain. Traditionally we would have a bunch of secret squirrel agreements in place and do a lot of board room negotiations. Now we do it together, in the open and we help everyone (engineering, hardware vendors, redshift companies and of course our customers).

With COMSTAR, anyone that wants to participate in COMSTAR and drive the value chain is welcome to. Solaris is a Storage operating system. The power of Solaris coupled with world class partners in an open, feedback intensive value chain drives the next generation of adoption. Together we accomplish so much more than we could if we were just doing it ourselves. Frankly with any of my products and projects, you can participate. You get the value of working with a world class engineering team and you get a chance to influence the next generation of design. Isn't that what it is all about? Four very large customers are either already deployed on COMSTAR or are making plans to deploy. One customer likes it so much they have agreed to purchase 50 Thumpers and a service contract prior to integration. They never would have seen it before it was in the OS if it hadn't been for the power of an open community.

Want to help drive IO for the next generation? Let's hook up...





« June 2016