Thursday Oct 09, 2008

A Time to Remember

A Time to Remember

“Only be careful, and watch yourselves closely so that you do not forget the things your eyes have seen or let them slip from your heart as long as you live. Teach them to your children and to their children after them.” - Deuteronomy 4:9 – New Internal Version

This is my last blog post as a Sun employee and is a great opportunity to remember. To remember the people that helped make Sun a wonderful place to work – more appropriately a wonderful family. It is also a time to refresh the memories of past lessons.

Let's start with the people. From the early days in Colorado Springs, folks like Dale, Mike, Lynn, Paul, Rick and Lou. Then there were people while I was helping with Sun Education. Dave, Howard, Siusaidh, and many others. That brings me to the past 11 years with Solaris. There have been so many great folks I could never mention them all. The great managers: Fred, Bev, Eric, Dave, Barry, Martie, Bijan. The stellar program managers: Gina, Reg, Eoin, Tina, Ann. And of course the amazing engineers: Sanjay, Jeff, Mark (at least 4 of them), Susan, James, Jean, Tony, Steve (x2), Moriah, Tom, and so many more. Then to OpenSolaris friends: Deirdre, Jim, Tim, Glynn, Theresa, Kelli and the community Ben, Dave and again many more.

It is because of all of these folks that I was able to do anything. They provided support, encouragement, kicks in the butt, prayer. They were all there to support the initiatives and move forward the relationships. I cherish these folks and the little bits they gave to me over the 14 years here at Sun.

This company has been home for over 14 years. My eyes have seen many things and there are many lessons that I plan to pass on to my children. Lessons like:
  • lend a hand to someone in need
  • stay what you mean and mean what you say
  • care about people and they will in turn care about you
  • a simple smile goes a long way to smoothing a difficult conversation
  • new ideas are always waiting to be found, you just have to find the friend to help uncover them
  • if you listen, your voice can be very loud

In all this has been a great time. Many wonderful memories, just as many friends and a very exciting future.

While leaving the group and the OS I love is not easy, the OpenSolaris Storage community in great hands, Peter, Lynn, Paul and Deirdre will serve it well. I look forward to the exciting innovations becoming realities and plan to watch very closely.

With this I bid you Peace,


PS: If any of you care to follow me and my thoughts my new blog will be

Friday Sep 26, 2008

Using remote storage with PostgreSQL

I stumbled upon an interesting blog post this morning. Jignesh Shah from the Sun ISV partner group blogged ZFS with Cloud Storage or Faraway Storage. In this post he describes how us uses some of the latest features of ZFS, like the ZFS L2 ARC and the Separate Intent Log to gain some performance when using remote storage. Seems like an interesting use of these ZFS features.

, ,

Thursday Sep 04, 2008

OpenSolaris Storage - no more white shoes

Traditions - an inherited, established, or customary pattern of thought, action, or behavior.

There are all kinds of traditions in families, cultures, religions and even companies. Some of my personal traditions range from making my grandmother's cinnamon rolls on Christmas morning, saying bed time prayers with my son, and writing monthly status reports. One odd tradition in the US is that women (and I guess men too) can not wear white shoes between the Labor Day Holiday and Memorial Day.  As I did a bit of research for this post I found that I'm not supposed to wear sandals after Labor Day either! Bagh ... For those who know me sandals or flip-flops are part of my attire most of the year. I have never understood the tradition of no white after Labor Day, but if guess there are lots of people who need rules and need traditions to survive in the fashion world. Just as it seems people have traditions in their personal worlds, so do they in the corporate/ work worlds. Some times these traditions are called best practices and other are just word of mouth policies.

Traditions, as great as they are, only go so far. They keep us connected to the past, but don't really enable a different future ... what about breaking with tradition, not being the new monkey, looking for new ways to do things ... I'm here to talk about the folks who are not afraid to break from tradition, the people who are willing to take a chance on something different. As I worked up the monthly report for the storage community It hit me that there are many folks who are using OpenSolaris at the heart of their storage environment or who are using new technologies to get beyond the limits of their traditions. There are customers like Joyent who were one of the first companies to put ZFS into production (before it was officially supported!). Guys like Nexenta who are creating new products with OpenSolaris ... or what about greenBytes? (I'm not sure what they guys are up to, but ZFS+ and a the tag line "Expect some magic on 9/15" makes me wonder...) There are also folks like OurStage and Liveammo that are putting the storage technologies of OpenSolaris into their production environment. ... all of these examples breaking the traditional image of storage ... what are your examples of breaking with tradition? If you will be in Santa Clara, CA on Sept 21,2008 come by the OpenSolaris Storage Summit and hear from some some of the people changing the storage industry.

So what's next? Have you checked out the latest project "Hadoop Live CD at"? or what about Project Celeste? These are some great examples of thinking about storing and retrieving "things" differently. I'm sure these project teams would love for you to stop by their project pages, take a look around and then start asking questions. Even better maybe you can jump in and help out.

With all this build up ;-) ... I will present the OpenSolaris Storage Community monthly report for August 2008.

The community also saw continued growth in message traffic (year over year) and in page views. Both of these stats showed monthly declines which seems to map to a general trend. I'll close this post with one final invitation to the OpenSolaris Storage Summit. Todate we have over 40 registered atendees, talks from Ben Rockwood, Mike Shapiro, Jeff Bonwick, and Bill Moore. Our attendee list represents a broad spectrum of OpenSolaris users and the after party looks to be a great time. If you happen to be in Santa Clara, CA on September 21, 2008 plan to stop by and enjoy the FREE event.

Tuesday Aug 05, 2008

Conversations ... Dave Stewart and Storage Summit

Present Conversation:

Last week I had the great privilege to interview the infamous Intel Dave ... a title that he really finds funny by the way. Dave was in town to visit family and he happened to be able to stop by the Sun offices in Broomfield for the FROSUG meeting. He gave a great talk on Intel's participation in OpenSolaris and how they are contributing. Before the FROSUG meeting I had the opportunity to sit down with him and chat about OpenSolaris and what it is like to be outside of Sun contributing to to this project. It was informative and the idea of giving people credit for their contributions resonated with me. We'll be trying to do this in the storage community. Watch the interview and then I'll be interested in your ideas on how to make people's contributions known. Drop a comment and let me know your thoughts.

Future Conversation

OpenSolaris Storage Summit

The OpenSolaris storage community in conjunction with the Sun's Open Storage Initiative are proud to host the first annual OpenSolaris Storage Summit, Sept 21, 2008. This summit will directly proceed the SNIA Storage Developer Conference, of which Sun is a sponsor and will be a great time for users and developers of OpenSolaris storage technologies to meet each others and hear about some cool happenings. So far we have signed up Mike Shapiro (of dTrace fame) and Ben Rockwood (Joyent dude and OpenSolaris evangelist) as keynote speakers. We are looking for other speakers and topic suggestions. So drop by the summit wiki, give us your suggestions and sign up to attend. The whole event is free of charge and we are planning a great party Sunday evening. Is you will be in the Bay Area on the 21st of September we would love to see you.

Monday Jul 28, 2008

Setting Up an OpenSolaris NAS Box: Father-Son Bonding - The Video

Updated!  Video transcript added below.

I just posted the write up on how my son and I built an OpenSolaris based NAS box. Now here is the video of our presentation at FROSUG.

21 mins.

Also available in iPod video format

My thanks to all who helped make this possible! 

Chase: You did an awesome job buddy! I'm so proud of you/ You took a huge step of faith to do this video and you did great. It is kinda weird to see yourself on video, but it is way cool. Great job, my friend!

Thanks to Paul, Damon, Lynn and Deirdre for all of their help making this video and the previous write up possible. "None of us is as smart as all of us." - Ken Blanchard

[NOTE: I'm reposting this entry to correct the mp4 version of the file. The original file disappeared some time yesterday. Sorry for any who tried to d/l and received a 404. --jc]

Video Transcript 

Jeff: Thank you very much, John. And thank you FROSUG for allowing us to come and show off what we've been up to for the last couple months. I’m Jeff Cheeney.

Chase: And I'm Chase.

Jeff: Outstanding. Well, we'll give a little bit of introduction on who we are.

Chase: Okay, so, I'm Chase.

Chase: I'm 11, and I'm in 5th grade, and this is my dad.

Jeff: And I am not in 5th grade, nor am I 11.

My job at Sun is the OpenSolaris community manager for the storage community. So I think about things about storage and I think about things like how to build a community and who would be interested in participating -- you know, contributing code or helping out with projects that we've got running, so that's part of what I do for Sun. And therefore, "the storage box" was an idea of ours in order to -- go ahead and advance that slide, Chase – [We] wanted to do something different, and we wanted to try some present technology and I wanted to see if we could do this as my son and I. So what else did we do ?

Chase: Um, we wanted a way to pass winter break and to learn about computers.

Jeff: Chase has this understanding that computers are for web browsing, playing games, occasionally doing spelling words-- you know, typing out spelling words a few times. And I decided that that was a little bit of limited knowledge, that we needed to teach him a little bit more. Show him a little bit of hardware, as well as a little bit of software. So that's what we're up to.

So, Chase, why don't you run us through the list of ingredients ? We could-- we could put 'em up over there as well as on the screen.

Chase: Okay.

Chase: So we have one Sun W2100z, which is Fred, big box. We have a SCSI HBA, 0ne 70-gig internal drive, one 140-gig internal drive, and we have one SCSI multipack. The multipack has six 40-gigabyte drives.

Jeff: So the thing I should mention about this is we decided we wanted to do something that we could just-- we're trying to do a demo here of what you might be able to find in your house-- Well, you may not find the multipack in your house-- but you'd find old hardware, find some stuff that you can cobble together. So what I did is I went in and raided the lab. So I went in upstairs, went into the Solaris engineering lab and said, "Okay, what's not being used ? What's on the racks that's not being used ?" So found an old 2100, had a gig of memory in it, scavenged out of some other ones that weren't being used, a couple of other DIMMs, so it could beef up the memory a little bit. Found an HBA that I could attach the multipack to, you know, so we've got a total of 200 gig on that multipack. Boy, that's real exciting. I got more hard drive space in the one computer at home than that multipack that weighs like 40 pounds and sounds like a jet engine. But nonetheless, we can do it all and we'll show you how it all works with what we've got.

The other components that were in our kit here, we're using the SXDE distribution of Solaris, so that's the Solaris Express Developer Edition. We're using the January '08 version, because that's what we had available at spring-- at winter break. And then the other thing I wanted to point out, and we'll mention this again-- there's a "getting started" guide that's available off of the Genunix wiki, and that wiki is where we do a lot of open source-- or, excuse me, the OpenSolaris collaboration. So that "getting started" guide was critical For our success and I wanna make sure we point that out. That's one of the key ingredients in our home NAS.

The other thing, I guess I should have mentioned a little sooner is, we were doing this as a recipe. We decided that people like recipes, they can identify with that format, there's ingredients, there's time, there's how you mix it all together, so we utilized that-- that methodology of this presentation.

All right, Chase.

Chase: Okay, so, we installed the hardwares first-- okay. And that took about 20 minutes or so, and then, um, we installed the SXDE, and that took forever. I was in bed when he finished it.

Jeff: Yeah, we got that started, it was about bedtime, we walked through the screens-- We'll talk about that in a minute-- But the full install takes at least a couple hours and it was getting a little past bedtime, so I finished that one up with him in bed and he got to check it out in the morning.

Chase: Then we configured Solaris, which took 40, 45 minutes.

Jeff: You know, the hope is with this presentation, as well as with that "getting started" guide, it won't take any of you 40 minutes to do the config. I was doing a lot of constructing 'cause we were going through this. I was also utilizing some of the different tools that we have in Solaris. If you're familiar with OpenSolaris, you're familiar with Solaris, you know the command line, it'll be a whole lot faster. But I was using graphical interfaces just so Chase had something interesting to look at 'cause it's kind of like a web browser, as well as just trying out some of the technologies that I hadn't utilized myself, so that's what we did.

Chase: And then we had Windows share the files from our computer to--

Jeff: The other way around, we shared it to Windows.

Chase: Yeah, nevermind.

Jeff: So we had to talk-- we had to configure Windows and yeah-- we'll get to that one. That one took a little while. All right, move on. All right, so here's the pictures of the hardware. Obviously, we-- like I said before, we used-- we used about 5 gig of memory. One of the big reasons we used that much memory is ZFS loves memory. When ZFS is active, it keeps a lot of pages in memory, so it-- that's what gives it some of its speed, but as well as it does its transactions in memory. So, because of that, we gave it a little extra memory. Okay, we gotta do your animations. Here we go, all right. All right, then what else did we do ?

Chase: We installed the SCSI HBA.

Jeff: Okay, why did we do that ?

Chase: Um...

Jeff: It's the HBA part, all right. Why did we put the HBA in ?

Chase: Um...

Jeff: So the HBA is how we connect it, the multipack, right ?

Chase: Right, yeah.

Jeff: So we had to add one of those in because we didn't have external SCSI on the 2100. Okay.

One more. You've got the flying connecter... And then we powered the whole thing up. And fortunately for us, we did all the installs of all the boards correctly and it worked ! And the thing to point out here was Chase was doing all this for the very first time. He'd never opened up a system before, he'd never seen a motherboard, never had seen a DIMM, um, never had seen an HBA. So it was a great opportunity to use grounding strips and teach him a little bit of electrical safety that we had-- at least I learned when I was in the university. Okay. So, tell us about OpenSolaris ? How hard was it to install ?

Chase: Not. All we did was follow the screens.

Jeff: So, like, what kind of questions did it ask you ? What did you have to do while you were installing ?

Chase: You made the users--

Jeff: Yeah, and before that ?

Chase: Passwords.

Jeff: Made some passwords, time zones...

Chase: Yeah.

Jeff: Did you have any trouble doing that ?

Chase: Nope.

Jeff: And that took how long ?

Chase: I don't know, five minutes ?

Jeff: Five, ten minutes, that was it.

Chase: Yeah.

Jeff: That's the point. Following the graphical installer was pretty trivial. We just walked right through it, didn't have any issues at all. So that was the easy part. The long part was getting it installed, and you didn't have to do that. All right, so now, we've got the machine built, we've got an operating system on it, what are we gonna do now ?

Chase: Configure Solaris.

Jeff: There you go, and what do we have to do ?

Chase: We have to read the "getting started" book.

Jeff: Again. Teaching-- read the directions. We're trying at least, to learn to read the directions. So, that's what we did. We read the directions, and then what did we have to do ?

Chase: Then we created the local users.

Jeff: Okay, what did we create ?

Chase: We created media.

Jeff: Right, why did we create media ?

Chase: So we could have all our files from our own computer on that one.

Jeff: Okay, so that's who's gonna own the files on the-- on fred, okay.

Chase: Then we have edmund ?

Jeff: We have edmund, right.

Chase: And root.

Jeff: Yeah, root gets created by default, so we already got that one. All right, so we got our users, and we got everything installed. Now what's up ?

Chase: Now you have to combine all the disks on the SCSI to make a ZFS pool.

Jeff: Right, so what does that mean ? What did we do ?

Chase: It means, um, you follow the screens again

Chase: To get to the place where you can combine the six disks in the SCSI to make a ZFS.

Jeff: Yep, so we took-- and the thing that was-- That was fun for me, was walking through the graphical interface that I knew existed, and the only reason I knew it existed was because I managed the engineer who wrote it at the time, but I had never played with it. And I kept hearing about how wonderful it was. So I gave it a shot, and you bring up a web browser, point it to a specific port and you've got a graphical interface. You can pick what kind of configuration you want for ZFS. It automatically defines all of the drives on the system, whether they're being used for something or not. You collect-- you select what kind of pool we wanted, so in our case, we created a single-parity raid-z. Added the drives, clicked "done," showed us the command that we could've typed, which is pretty straightforward as well. And how fast did it create this ?

Chase: Pretty quick.

Jeff: How fast ?

Chase: I don't know.

Jeff: Like-- that fast.

Chase: Yeah.

Jeff: You didn't even have to think. It really-- it is snap-a-finger fast. All right, so we created the pool…

Chase: Right.

Jeff: And then what are we gonna have to do ?

Chase: The file systems.

Jeff: Okay, what are the file systems for ?

Chase: So we could share.

Jeff: Right, and that's where we can put different kinds of files, right ?

Chase: Right.

Jeff: So what we did is we went through and created three different file systems, all in our pool. We created-- we'll show these in a couple minutes-- se created one for our movies, one for our music and one for our pictures, xo that's what they are. And we set some different properties on those as well, so knowing that our music might be compressible, we turned compression on on that file system, but we didn't do it on the movies because movies are typically pretty well-compressed and so it would be just wasted cycles to try to recompress the music. And so we did some tweaking like that of the different file systems. And we did-- again, we did all that through the browser interface. That was the time where you actually had to do some typing.

Chase: Yes.

Jeff: Chase is still learning how to type, so this one took us a little while. So what did we do ?

Chase: We configured CIFS-- c-i-f-s ?

Jeff: CIFS, right.

And what is CIFS used for ?

Chase: Isn't it used for sharing ?

Jeff: That's right, it shares our files from our OpenSolaris box to...

Chase: Our regular computer.

Jeff: Our regular computer-- In this case, it's a Windows box. Don't you love this ? So, yes, we shared it to our regular computer. All right, and so how do we do that, we turned it on...

Chase: Yeah.

Jeff: So we used what command ?

Chase: Um...

Jeff: You can read it off the slide.

Chase: "S-v-c-a-d-m enable."

Jeff: Right, so we enabled that service, the svc service, we enabled it, and then we did the join. So the join step right in there is-- at our house, i've created a workgroup for all of our computers and our workgroup is called "central," that was all that we had, so I used “smb join,” which is the-- Smb is the file-sharing protocol administration, I'm joining the centralized-- the central workgroup. So that was a critical step in that whole process, To make sure that we make-- that we could share our files in that workgroup and that we set up the CIFS server to be in workgroup mode versus another mode, like active directory, okay ? So then what did we do ?

Chase: We shared the file systems ?

Jeff: Mm-hmm. And how did we tell ZFS that we wanted to share them ?

Chase: Um, we did "sharesmb" and then we did the name of the file.

Jeff: Right. So this is one of the tricks that we had to pull out of the guide that I wouldn't have figured out before and honestly, this is one of the places where I really struggled with the Windows configuration and getting Solaris to work. This is now in the "getting started guide" as well as on this presentation and it'll be on my blog in a couple days. but-- as well as Doug McCallum put this on his blog. So "sharesmb" is the command-- is the subcommand to ZFS to say, "I wanna share this file with the CIFS protocol, I wanna share it as an smb share versus an nfs share." And then what you can do is you can name that share So that when Windows is browsing your OpenSolaris box, you'll have a name associated with that share. What was happening is, without that specific name in there-- so in our case, the very first line, "movies"-- the share was being-- it was being exported as the "zpool" name. So in our case, "mediapool," with an underscore and then the file system name. So the "mediapool, underscore, movies." And it wasn't obvious because when you went into the Windows share browser, it didn't show up that way, and so I really struggled with, "How do I configure this ?" It took me a while to read through all of the discussion lists, which are, you know, as you guys all probably know, it's very rich with information.

So found it in there, dug it up, It's now in the-- excuse me-- "getting started" guide. That was one of those critical pieces of information that took a while to figure out. So we share our three different file systems, gave 'em unique names, And we've got some mixed case in there as well, so we can share the different case sensitivities. And then we had to make our files writable to... Media, our local user. We had created these all as root so they, by default, get mounted and shared as root so we had to go ahead and change the permissions on those. So we did that.

Jim: So Jeff, is that--

Jeff: Yeah, just one second, Jim. If we could get a mic to you so you can ask a question so we can get it on tape, that would be great.

Jim: So, jeff, that command, I kind of missed it. So it's not case-sensitive, I guess ? The ZFS share command ?Yeah, because if you have the movies with a capital "m" and then you have a lowercase--

Jeff: It is case-sensitive. So you can share it with different cases.

Jim: Oh, okay.

Jeff: And actually, you bring up a great point there, Jim, and I should have put this on the slides and I'll make sure I update my blog with this. There is another ZFS command that you need to use to tell ZFS to accept case sensitivity. So ZFS said "case-sensitive equals on," with a pool, and then it'll be a-- that property will be inherited across all of the-- across all the file systems. Thanks for that reminder, jim.

Great, now like I said, the tricky part, getting Windows-- getting the “regular” computer to talk to the OpenSolaris computer. We're gonna change that, Chase. OpenSolaris is the regular one, that's the irregular.

Chase: I like the regular one.

Jeff: So it's fred and not fred. So, again, this was tricky. This took-- once we figured it out, once we got the guide, it all goes fairly quickly, but learning, this was a big learning curve for me. I'm not a Windows guy. I'm a Solaris guy. I've been doing Solaris for 20 plus years, and-- Or, right at 20 years. So what we had to do is, in the Solaris developer express build that we had, our CIFS server on OpenSolaris didn't export things in a browsable mode, so as some of us are accustomed to in Windows, you can go to a machine and browse that machine and you can see all of the shares that it's offering out.

In the build that we're currently running on this machine, that didn't work, so you had to explicitly-- you had to explicitly mount a drive. So you had to say, "well, I mount-- in this case, drive "z"-- and I want it to be this explicit share," and you have to name it. In subsequent builds of OpenSolaris, the builds that are available now that you can go out and download, browsing is available, so you can browse and mount those shares with just a single click.

So in this-- this is the case where you'll see, Fred is our machine name and our share, just as we talked about before, it's capital, It's Movies with a capital "m." The other big thing, it took us a while-- It took me a while to figure out because I didn't read-- I didn't read the guide as thoroughly as Chase did and he was in bed at that point.

When you go to mount that share, the access privileges have to match the OpenSolaris box, not your Windows box. So in this case, I was trying to mount these shares as "owner," and "owner" doesn't exist on my OpenSolaris box. So the critical piece here is that you'll notice on that mapped network drive-- this is Windows XP -- I connect as a different user. You click that different user and then you put the credentials in, login and password, for the OpenSolaris machine.

So in our case, we had created an account media, that's what we're gonna be mounting the shares as. So that's another little trick. It escaped me for the longest time and it's an absolute critical step. And with that, we're sharing files. We're reading and writing files back and forth between the machines, and we're up and running. And that really-- I try to distill all of my learning down into these simple, four, five slides. And that's how straightforward it is once you get through the learning curve. So hopefully, i've done some things to help you all get through the learning curve and you can try this out as well.

So just like any good recipe, you gotta look for variations. How do you vary this recipe ? How do you make it tailored to your needs ? So one of the things we can do--

Chase: Get bigger disks.

Jeff: Get a lot bigger disks. So we each have two half-terabyte drives now that we're gonna be putting in the box, we're gonna try sata connections. We're gonna see if we can get the onboard sata on the w2100 to talk to those-- to talk to those half-terabyte drives, so that's gonna be our next project.

One of the other things we wanna do is, OpenSolaris has embedded in it the protocol, "icap" and in conjunction with icap, we have the support to be able to catch file events. So why do you wanna do this ? Well, antivirus is a great example of this. So there's a program out there, “clamav"-- and it's an open-source program and you can find it at this link-- that will scan files on OpenSolaris, it'll scan Windows files on OpenSolaris for known viruses. You can download virus definitions and it will be automatically scanning. The really cool thing that happens is, every time ZFS gets a write or it gets a new file create, ZFS stops the operation, hands it over to the scanning engine and lets the scanning engine proceed first. And it won't-- it won't commit that file to disk until it knows that it's clean. So it's a really cool thing-- or cool feature, because now I can run virus-scanning as files are being created on my OpenSolaris box, which has a whole lot more processing power than my "irregular" computer. And I don't have to worry as much about the viruses that somebody might be downloading onto the computer.

Chase: It's called e-mail.

Jeff: Yeah, e-mailing viruses, that's right. One of the other things we wanna do is we wanna take this box and instead of having it sitting in a closet somewhere, I'm gonna put it somewhere else in the house. And with sunray technology as well as with VirtualBox, I want to have an “irregular” box inside of Solaris. So if the kids wanted to play their games on Windows, we could do that, we could use our VirtualBox software, which, if you guys don't know about VirtualBox, we probably should come back and have a fun talk about that because it's just a lot of fun. It's the vmware for-- that actually runs on OpenSolaris, OpenSolaris is the host.

So those are a couple of things that we're thinking about doing, and obviously any other things that might be in your imaginations as well. All right, and we've gotta give credit where credit is due.

Jeff & Chase: The bibliographies.

Jeff: He was the one who told me, "you gotta have a bibliography." He's been writing too many research reports. So here's our sources, these are the places that we have referenced heavily to get this working.

The CIFS server project page on OpenSolaris was a great resource for us. Doug McCallum, I'll mention Doug again, he has a wonderful blog that is very detailed, full of rich reference material on how to use the "share adm" command. Just a fabulous reference,

And then those two wiki pages that we referenced as well. And I probably should have put on there the OpenSolaris storage discuss forum. That's where all of the discussions, the tips and tricks for this project are going on, so that's OpenSolaris. The storage discuss on OpenSolaris.

So that's it ! We're done. In all, it was a fun thing. Did you have fun ?

Chase: Yeah.

Jeff: You learned a couple things ?

Chase: Yeah.

Jeff: You might wanna play with it a little bit more ?

Chase: Mm-hmm.

Jeff: All right, all right. So with that, if you guys have got any questions, please do ask for a microphone since we are trying to record this, it is live. But that's what we did, hope you enjoyed it ! Great ! Thank you.

Monday Jul 21, 2008

Pages of interest - 21 July

Elektronkind » OpenSolaris 2008.11 - A Preview For The Storage Admin - I missed this page last week, but wanted to make you all aware of an interesting review of OpenSolaris from the perspective of a storage system administrator. There is good coverage about the OpenSolaris storage features of ZFS, iSCSI, CIFS, COMSTAR, NDMP, AVS and others.
The Blog of Ben Rockwood » : Explore Your Storage with FileBench - This blog was posted a few days back and gives a nice overview of FileBench and talks about how to create various workloads to determin the performance of a system.

I also wanted to point out:
mega_sas project page now live - /devices/pseudo/bitbucket@0:bitbucket - the mega_sas project is now live on OpenSolaris. This is an open source implementation of the driver for the LSI MegaRAIS SAS Controller. The project page has lots of information on where the project team would like to head next and is looking for participants.

[4-Aug: This post has been correct to give Ben Rockwood credit for the FileBench post.]

Monday Jul 14, 2008

Laptop and retired I/O devices

FMA and my laptop ....

Last Thursday was not much fun. The “reduction in force”, more appropriately called lay-off, was a challenge. Good people lost their jobs and those of use remaining are trying to figure out how to still get our jobs done with out some key support folks.

Well coincidently with the RIF my laptop started mis-behaving. Upon reboot Friday (the day after the rif) I was greeted with this error message:

NOTICE: One or more of your I/O devices have been retired

Great, now did my laptop get RIF'd?

Actually not. Thankfully!

What happen is that FMA detected too many error with my on-board ethernet driver and it disabled the faulty component. How cool is that?!? The operating system on my laptop detected a faulty component and disabled so the processor didn't have to keep dealing with the interrupts.

This is way cool, but it was a bit difficult to figure out what was happening.

To save some of you the pain of debugging this condition, here is a quick post on how to determine what's happening and how to fix the condition.

First, let's check to see where the error message was coming from. A quick search at resulted in a link to retire_store.c:241. Here quick look around the source let me know this was from the fault management portion of OpenSolaris.

To see the fault, use fmadm with the faulty sub-command:

$ pfexec fmadm faulty
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Jul 11 06:30:19 6dc01480-53e3-6046-92c5-b88ea74e17af  PCIEX-8000-0A  Critical 

Fault class :
Affects     : dev:////pci@0,0/pci8086,27d0@1c/pci1179,1@0
                  faulted and taken out of service
FRU         : "MB" (hc://:product-id=TECRA-M5:chassis-id=Y6071991H:server-id=yoyo/motherboard=0)

Description : A problem was detected for a PCIEX device.
              Refer to for more information.

Response    : One or more device instances may be disabled

Impact      : Loss of services provided by the device instances associated with
              this fault

Action      : Schedule a repair procedure to replace the affected device.  Use
              fmdump -v -u  to identify the device or contact Sun for
With the device path I checked to see if this path corresponded to any known devices:

Check that out ... the e1000g0 is the driver for my ethernet!

$ ls -l /dev/\* | grep "pci1179,1@0'
lrwxrwxrwx 1 root root 54 2008-04-24 16:32 /dev/e1000g0 -> ../devices/pci@0,0/pci8086,27d0@1c/pci1179,1@0:e1000g0

but that device is not available:

ls -l /devices/pci@0,0/pci8086,27d0@1c/pci1179,1@0:e1000g0
/usr/gnu/bin/ls: cannot access /devices/pci@0,0/pci8086,27d0@1c/pci1179,1@0:e1000g0: No such device or address

Knowing that this was the device that was not working I had to correct the fault. Here use the fmadm command again with the repair sub-command and the path the to device.

$ pfexec fmadm repair dev:////pci@0,0/pci8086,27d0@1c/pci1179,1@0
fmadm: recorded repair to dev:////pci@0,0/pci8086,27d0@1c/pci1179,1@0
$ ls -l /devices/pci@0,0/pci8086,27d0@1c/pci1179,1@0:e1000g0
crw-rw-rw- 1 root root 225, 1 2008-07-14 13:40 /devices/pci@0,0/pci8086,27d0@1c/pci1179,1@0:e1000g0
Now to reenable the device and replum my network
$ pfexec ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
	inet netmask ff000000 
wpi0: flags=201004843 mtu 1500 index 2
	inet netmask ffff0000 broadcast
	ether 0:18:de:6a:9e:5a 
ip.tun0: flags=10008d1 mtu 1402 index 7
	inet tunnel src tunnel dst
	tunnel security settings  -->  use 'ipsecconf -ln -i ip.tun0'
	tunnel hop limit 60 
	inet --> netmask ffffffff 
lo0: flags=2002000849 mtu 8252 index 1
	inet6 ::1/128 
ip.tun0: flags=2204851 mtu 1480 index 7
	inet tunnel src tunnel dst
	tunnel security settings  -->  use 'ipsecconf -ln -i ip.tun0'
	tunnel hop limit 60 
	inet6 fe80::a07:fbeb/128 --> fe80::1 
ip.tun0:1: flags=2200851 mtu 1480 index 7
	inet6 2002:8192:117b:1::a07:fbeb/128 --> 2002:8192:117b:1::1 
$ pfexec ifconfig e1000g0 plumb
$ svcadm restart nwam

Everything is up and running again. It's pretty cool to have enterprise features like FMA on my laptop, but being a simple guy it's good to know how to just get up and running again.

Now I'll keep monitoring my ethernet driver and see if it keeps acting up ...

Once again, the power of an enterprise class operating system on my laptop. This is so cool. Thanks to the FMA team for your great work!

Friday Jul 11, 2008

Happenings in the OpenSolaris Storage Community

Happenings in the OpenSolaris Storage Community

This has been a crazy week. I've been running around trying to keep from having too many tasks fall on the floor. Unfortunately I don't think I'm doing very well. I'm sure way too many things didn't get done ... but let's quickly catch up on what good things did happen:

With all this going on, and having some good friends no longer working at Sun –enjoy retirement Mike, I guess it has been a pretty full week.

Until next time ....

Friday Jun 13, 2008

COMSTAR - Server = Storage Array??

Over the past few weeks information about a great new storage feature in OpenSolaris has been trickling out. The feature is COMSTAR (COMSTAR = COmmon Multiprotocol SCSI TARget). To learn more about COMSTAR the OpenSolaris project page include a number of videos and an outstanding administration guide.

I personally had the chance to play with COMSTAR a few weeks back and found it simply amazing. It was so cool being able to create a ZFS volume and share it to a windows host in around 10 minutes. I wrote up my experience in this presentation. There is a chance the presentation as given to the Front Range OpenSolaris User group may soon be a video.

Some other interesting reading about COMSTAR are:

There was also the press coverage:

I'm sure the excitement is just getting started and I really look forward to how this technology will be used in a future product ...

Friday May 30, 2008

In their words - Amit Vashi, Qlogic

The next installment in our ongoing series of interviews is now live on OpenSolaris. The interview I did with Amit Vashi covers why Qlogic is participating in the OpenSolaris Storage Community and where they are finding value.

Radio May 2008 Interview with Amit Vashi, Vice President, Qlogic

Check it out ... and let me know if you have ideas on who should be featured next....

Wednesday May 14, 2008

more home NAS and ZFS boot!

Things have been crazy over the past few weeks. There have been so many wonderful things going on (like Community 1 and the OpenSolaris 2008.05 launch) and I've been so busy I have neglected my blog and audience.

Many of you took some time to watch the video of my experience building a home NAS box. If you are interested in such things I would highly recommend taking a look at Simon Breden's blog on his experience. Not only does he cover the configuration, but also the rational on why he chose his hardware and ZFS.

Speaking of ZFS, I would like to publicly congratulate the ZFS boot/ install team for their recent putback of the ZFS boot project. Lori, George, Eric, Mark, Lin, Eric and others put in well over 10,000 hours of effort (much of which was in the last 5 months) to make this project a reality. Back in build 88 the ZFS boot project was putback and this enable SPARC systems to use ZFS as the root file system. Then just this morning the install portion was putback into build 90. Again this putback enables systems to be installed with the legacy installer and use the features like LiveUpgrade and JumpStart for installation. The ZFS boot page has an FAQ and other useful documents on this project. Way to go team!!!


Tuesday May 13, 2008

Storage Community Update - April

April was an exciting month for the OpenSolaris storage community. We updated the community web site with new pages and saw a large increase in web page views. We also approved a new project and other projects made good progress. We also interview 2 community members.

View all of the updates on the April What's Happening page ... and if you have more news, please update the wiki page.

Thursday May 08, 2008

OpenSolaris community - an introduction

Deirdre just posted some great videos from the OpenSolaris developer summit. This was a great weekend and it was fun getting to see the faces behind the messages on opensolaris-discuss.

Thanks for the great videos Deirdre!


Tuesday Apr 29, 2008

Developer Summit & Storage


Thanks Jesse for the great graphic for the upcoming developer summit. 

If you are interested in OpenSolaris storage and the latest happening, stop by our chat on Sunday afternoon. Unfortunately our 11 year old mascot will not be there, but I can bring autographed pictures.

Monday Apr 28, 2008

OpenSolaris Storage - in their words

Today we are kicking off a new part of the OpenSolaris Storage Community. -- the Community Spotlight. The community spotlight is a place where we will feature the contributions and successes of OpenSolaris Storage.

Checkout out our first 2 interviews:

Radio Interview with Evan Powell, CEO Nexenta
Radio Interview with Gregory Perry, CEO Liveammo

Do you have a story to tell? I would be interested in hearing it...




Here I am writing about the OS I love. Trying to show other people how cool the innovations are in [Open]Solaris.


« July 2016

No bookmarks in folder

OpenSolaris Storage Links