Friday Oct 23, 2009

Using the 7000 BUI to document script generation

I'm old skool enough that I prefer a CLI versus a GUI for administration tasks. I've got that disclaimer there because I happen to like web browsers, GIMP, and even WireShark at times.

Given a choice between being able to ssh (or even telnet) in to a box versus going to a web based interface, I'll almost always chose the ASCII interface. The exceptions would be when I don't have a manual and need to do something fast. If I'm not in a hurry, I'm willing to learn by trial and error.

I've been running some tests on a 7410 and I need to be able to restart NFS. I've been using the BUI because I really find it intuitive. For example, I needed to bring up another interface on a backend network and it was very simple to go to Configuration -> Network and just click my way through it. I also liked that my routing tables were set up correctly. I had just fought that in a similar situation on a v20z running Fedora 11.

I like being able to click on the Device and getting the relevant datalinks and interfaces. I like the apply changes implementation. I like it checks for uncommited transactions if I try to leave the current entry.

But what I really like is a very understated correlation between the BUI and CLI - the BUI documents scripting actions you can do in the CLI. Likewise, if you understand the CLI commands, you can quickly navigate to a BUI counterpart.

So my goal was to be able to restart NFS from the CLI. Rather than picking up a manual, I looked at where I was at in the BUI, which was Configuration -> Services. And to restart it, I selected the circular arrows to the left of the NFS Data Service.

This was 99% of the equivalent command:

washdc:> configuration services nfs 
abort    commit   done     get      script   show     
assert   disable  enable   help     set      

I was looking for a "restart" option and I realized that the button must be a combination of "disable" and "enable". Sure enough, this:

washdc:> configuration services nfs disable
washdc:> configuration services nfs enable

was functionally equivalent.

I find that every time I interact with either the BUI or CLI, I find something else to remind me of the tight integration and also to appreciate them both being designed at the same time.

Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Friday Mar 13, 2009

Trip report for OKCOSUG - Scott McCord, Sun Storage Architect

I went and saw a neat presentation by Scott McCord at the Oklahoma City OpenSolaris User Group meeting last night. He mainly talked about our Unified Storage 7000 products, building up the rational for why a customer would want to use them.

The timing was awesome as it dovetails into my exploration of the systems. One of the key take aways for me is that the morphing of OpenSolaris offerings into Unified Storage was not a small step. Yes it is built on the shoulders of everyone who has ever worked on Solaris, but there is a lot of small and subtle changes which really transform a collection of hardware into a powerful appliance.

I'm going to highlight what I learned, which may or may not dovetail into what Scott presented.

Write Optimized SSD

I had kept on wondering why Sun wasn't exploring NVRAM and the answer is Write Optimized SSD. Sweet! And then Scott showed a slide of a clustered 7410 and the Write Optimized SSD wasn't in the head, it was with the disk shelves. The implications were immediate:

  1. You don't have to send WRITEs over the heartbeat to your cluster partner. This has a huge impact on the call path of WRITE operations.
  2. Since you don't have to mirror your partner's WRITEs, then you do not have to effectively half your available stable storage. I.e., you get to use what you paid for!

No licenses

This went over well in the room and I think might actually be one of the hidden definitions of OpenStorage. :-> It is simple, you buy one of our appliances and everything that is supported by the hardware is immediately open for you to start deploying. You don't have to contact a sales rep, you don't have to rush through a PO because your management chain decided to start using your NFS appliance as a CIFS appliance as well.

And when you download the Sun Unified Storage Simulator, you can play with all of the features right away. You don't have to ask for permission.


This is basically a fancy way of saying that we have a graphical frontend to DTrace. And DTrace is the coolest customer support tool ever. It sounds weird, but let me explain.

DTrace - the coolest customer support tool for the kernel developer

Before DTrace, if a customer was hitting an issue that didn't present a core dump and static trace points weren't capturing the data, the only solution was to prepare a custom kernel to ship to the customer. This would entail the developer guessing at what data they wanted to collect, unit testing, and passing the resultant kernel off to QA. They would then do unit testing, perhaps try to recreate the customer environment (and a shout-out goes to Bill Snider who was really good at doing this when I was at NetApp), and then do regression testing.

The kernel would then go to the customer. And they would then normally run some of their own regression testing. They would then put it on the production system. If you were lucky, you got the data you needed. If you were unlucky, then you either had to roll another version (with the customer losing faith in your abilities) or you got a core dump. And again, you might still get lucky and catch the problem in the core. But you really had an irate customer either way.

With dynamic tracing, you remove that whole cycle of verifying a custom build. And remember, that cycle could be several weeks long! Instead, you piggyback right off of a well tested product and the resulting quality. You focus right in on the issue at hand. And even better, there are some pretty sharp sysadmins out there who can run DTrace on their own!

DTrace dispels FUD

Along these lines, I asked Scott to include the video from Brendan Greg's blog on Unusual disk latency (video is here).

I'd heard out at Connectathon 2009 that our appliances would fall over if you shouted at them. I knew that our competitors were going to use this as FUD. So I was really happy to find Brendan's openess to the issue.

But don't lose sight of Brendan's real message here - Sun has the only tool dynamic enough to measure the impact of shouting at a disk. Any competitor's disks are going to react the exact same way in the face of a shout. But can they measure the impact without that cost of developing special static trace points?

My point to Scott is that the video is exciting to watch (thanks to Brendan's upbeat personality) and drives home the point of what DTrace and Analytics brings to the table.

These are appliances, not general purpose servers

Someone in the audience stated that they just got a Thor box on a Sun Try and Buy program and wanted to know if they could convert it directly to a Unified Storage product. The short answer is to turn that Sun Fire X4540 Server back in and get a Sun Storage 7210 Unified Storage System out for an evaluation.

Scott didn't blindly recommend this, he asked the audience member what they were doing and the answer was it was intended to be a CIFS Home-directory server. If the answer had been along the lines of they wanted to run something on the box, then the recommendation would have been to stick with the Thor.

The point here is that the Unified Storage is optimized as an appliance. We aren't going to let you run a program on it. The focus is to leverage the reliability of OpenSolaris, the rock solidness of the underlying hardware, the experience we've had in configuring ZFS, tested hardware configurations, and the ease of use of the BUI.

We haven't moved away from our stance that we have great OpenStorage systems that allow you to run your application right on the server. Instead, we are focusing on taking that same hardware and software base and turning it into an appliance tailored to serving up data. Our value add is the way you interact with the appliance, the testing we put into both the software and hardware configurations, etc. I.e., we make sure the configurations work right out of the box. And in a stroke of genius, we make sure that the configuration of the box is simple.

Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Sunday Mar 08, 2009

Taking the simulator out for a run

Okay, I took the Sun Unified Storage Simulator out for a light walk the other day. And I made sure to avoid the documentation. Now I want to see what I can do with the documentation!

Fun Fact #1: Did you know that the CLI supports tab completion? Way cool.

So, what do I need to do to see all of the network interfaces?

snarky:> configuration
snarky:configuration> net
snarky:configuration net> interfaces show

    e1000g0 up     ip    e1000g0     Untitled Interface
    e1000g1 up     ip    e1000g1       outbound

Too long, can we do better?

snarky:> configuration net interfaces show

    e1000g0 up     ip    e1000g0     Untitled Interface
    e1000g1 up     ip    e1000g1       outbound

I tried to do this with slashes first, like a path, but I should have known better because we use a 'show' command, not a 'dir' like in the LOM interfaces.

Now, we get to see some of the power of this CLI! First, we can add ssh keys.

snarky:> configuration preferences keys
snarky:configuration preferences keys>

I was able to tab complete my way with each sub-context.

snarky:configuration preferences keys> create
snarky:configuration preferences key (uncommitted)> set type=DSA
                          type = DSA (uncommitted)
snarky:configuration preferences key (uncommitted)> set key="copy from your"
                           key = copy from your (uncommitted)
snarky:configuration preferences key (uncommitted)> set comment="thud-key1"
                       comment = thud-key1 (uncommitted)
snarky:configuration preferences key (uncommitted)> commit
error: An unanticipated system error occurred: Illegal character in key: ' '

This may be due to transient failure, or a software defect. If this problem
persists, contact your service provider.
snarky:configuration preferences key (uncommitted)> set key=""
                           key = (uncommitted)
snarky:configuration preferences key (uncommitted)> commit

The CLI will deny a command if it has invalid input. This feature really drove me to understand what was going on.

Anyway, the biggest stumbling block I had here was in what went into the 'key' property. Sometimes I am guilty of being too terse and connecting all of the dots in my head. I think the documentation here has done that as well. I really worked my way around the CLI and the BUI (Browser User Interface) trying to figure this one out. Along the way, I found that as I learned the CLI interface, the BUI interface started to make more sense. And that in turn made the CLI more natural. I liked the positive feedback loop here.

The BUI finally clued me in on what to do here:

ssh public key

I needed to be entering the contents of my '' file, which I previously I had generated with 'ssh-keygen'. Once I figured that out, and that it was easier to enter with the CLI (using a browser to add a long key is normally error prone because of automatic insertion of spaces), I was good to go:

[thud@adept ~]> ssh root@snarky configuration net interfaces show

    e1000g0 up     ip    e1000g0     Untitled Interface
    e1000g1 up     ip    e1000g1       outbound

And now, can I do the shares?

[thud@adept ~]> ssh root@snarky shares select default select jaloppy get sharenfs
                      sharenfs = rw,anon=braves,

I struggled for a bit on how to chain the 'select', I wanted to do something like a shell script '(cd dir1; ..)', but I did find this shining example on how to automate ssh scripts to our boxes: Snapshots with PostgreSQL and Sun Storage 7000 Unified Storage Systems.

I'll save for another day creating a script to drill down and get all of the NFS shares from a Sun Storage 7000 Unified Storage Systems.

The CLI for the Sun Storage 7000 Unified Storage System actually blows away my experience with the NetApp CLI!

  1. There is no telnet access - you have to use ssh.
  2. Command structure for CLI matches that for the BUI - learn one and the other will follow.
  3. The 'help' command is always available and always context sensitive.
  4. Stringing together complex commands is as easy as removing line returns. I.e., don't hit enter, just keep on typing!

Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Wednesday Mar 04, 2009

So what didn't I like about the CLI?

I got an email asking me what I didn't like about the Fishwork's CLI (see Impressions of Sun Unified Storage Simulator). I thought I would answer here.

I'd say the main complaint I had with it would be I couldn't issue 'ifconfig' or 'share'. I was trying to help someone over on a Linux mailing list with an issue (you can find the thread here -- Permission problem with NFSv4 mount and [Fwd: Re: Permission problem with NFSv4 mount]) and I wanted to just dive right in like it was any other OpenSolaris box.

It was actually easier for me to use the web interface than the CLI. Was the CLI that hard to use? No - just different. It was easy enough to figure out how to move about in with a couple of minutes of experimentation. I never pulled out any documentation.

So, it would be easiest for me if the CLI had normal unix commands. That doesn't mean that it would be easiest for everyone. This new product was the perfect time to change the CLI - you do not want local access on a appliance. Within pNFS, we are struggling with having local access for the MDS and DSes. Rob Thurlow is busy working on Proxy IO in case the MDS needs to read/write directly to the DSes (it also is a cheeky way to provide NFSv3 and CIFS access to the community).

So given that you want to deny local access, why not change the interface? And if you are going to change it, why not start off from scratch?

The whole GUI design reeks of object oriented design. Why not also have a CLI that does that?

Hmm, I'm not hitting specifics, which tells me a couple of things:

  1. I missed some common commands.
  2. I was quickly able to figure out how to use the new CLI.
  3. My lack of real frustration tells me that the ease of use outweighed familiarity.

I can't stress that I went into using the sim convinced I was going to hate it. I liked the NetApp sims, in part because I worked on it. But the NetApp sim was in no way as easy to configure or deploy. It took me a couple of minutes to get it up and running on the VMware network and only a couple of more to get it running across the physical wire. In the meantime, I had Debian Lenny not install on both a VMware machine and a mac mini. I finally got it working on an old laptop.

I walked away from using the simulator very happy with my experience. I think I was able to replicate the issue that a customer was seeing - I'm still waiting on a reply from him to confirm that. I'm confident that I can use it to troubleshoot other issues that might be reported on NFSv3 and NFSv4 for the Sun Storage 7000 Unified Storage Systems.

I also know that a lot of effort was put into observability for these boxes - just look at Bryan's entry on Eulogy for a benchmark or Brendan's entry on Unusual disk latency. My take away from both of these is that we can now measure things that other vendors can not. Before Dtrace, you had to systematically add trace points, recompile, QA, and then ship new bits to customer sites. And they may have had to QA the incoming bits. Now you can just start asking the interesting questions right away and getting answers.

Without delving too deeply, I expect all of this to be more integrated with the new UIs. I suspect there is a power here. I'm pretty sure I can quickly dig through this stuff on the web interface. And I'm just as confident that the CLI has the same power, but it just isn't my parent's '>' prompt...

Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Impressions of Sun Unified Storage Simulator

So I have to admit that I hate GUI interfaces to appliances and I love simulators. At NetApp, I loved the really streamlined console you had for a filer and hated FilerView. I would roll my own Perl scripts to maintain a large pool of filers when I was a filer admin for both Corporate and Engineering IT over there.

And when I worked for NetApp's NFS team, I loved the sims we used for quick and painless testing.

So I approached the Sun Unified Storage Simulator with a bit of trepidation. It turns out that you can ssh into the box, but doing remote administration that way is no where near as nice as the NetApp filer console was three years ago. (Note: I have no experience with the GX series console.) Update: I've quantified my views on the ssh access over at So what didn't I like about the CLI?.

The better news is that the remote http access is way better than FilerView. I was able to quickly understand the relationship between projects and shares. I was able to quickly configure a complicated share on my storage. I loved it.

I also loved that it worked straight away, right out of the box. I happened to already have VMware Workstation 6.5.1 on my home desktop, so I was up and running. I was a little disappointed that it only supported local access on the desktop, until I realized I had 3 other network interfaces I could configure. Before I knew it, I had mounts going to it from remote computers.

I really found the GUI to be intuitive and I only struggled for a bit wishing for a CLI. I also used the minimal CLI to gather some data on the share. It reminded me a lot of the way our LOM managers work.

The other real plus in comparing this simulator versus the ones NetApp provides is that I don't need any license keys. Sweet!

Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Tuesday Mar 03, 2009

Sun Unified Storage Simulator

I'm trying to help someone on with a NFSv4 issue between a Linux client and a Sun Storage 7410 system. I don't have my hands on a real 7410, but I found this great resource for letting me simulate one: Sun Unified Storage Simulator.

It downloaded easy and I'm about to configure it...

Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Thursday Oct 23, 2008

Just saw an interesting integration float by

I'm being a complete OpenStorage fanboy, I just saw the integration notice for 6760398 Moving NDMP to open source go by and without even knowing exactly what it entailed, I was happy.

So happy I downloaded the Mecurial source onto my Linux server to make sure I could already see the change:

[thud@adept onnv-gate]> hg history | more
changeset:   7917:5c4442486198
tag:         tip
user:        Reza Sabdar <Reza (dot) Sabdar (at) Sun (dot) COM>
date:        Thu Oct 23 11:42:48 2008 -0700
summary:     6760398 Moving NDMP to open source

Another big step for OpenStorage:

Author: Reza Sabdar <Reza (dot) Sabdar (at) Sun (dot) COM>
Repository: /export/onnv-gate
Total changesets: 1

Changeset: 5c4442486198

6760398 Moving NDMP to open source


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily



« July 2016