By Nalini Kotamraju on Sep 14, 2007
Nalini Kotamraju is a user researcher in xDesign, and a PhD in Sociology. She has a penchant for research methods and telling it exactly like it is.
Years ago, shortly after I joined the Software User Experience Group (xDesign) at Sun, my manager asked me whether I would be willing and able to conduct a usability study of a new CLI for one of our software products, Sun Cluster. I, the ever eager new employee, promptly responded yes, that I'd be thrilled to do such a study. I then withdrew to my desk, and typed "CLI" in Google to figure out what it meant.
CLI stands, of course, for command line interface, which is a way to interact with software or an operating system. Once I met with the product team and had my first look at the CLI, I understood why my manger had wanted to feel out my reaction to this kind of study. By the time I joined Sun I was a veteran at usability studies, having led many a user through a graphic interface in paper prototypes or interactive mock-ups (usually web sites of now-failed dot.coms). Testing the intuitiveness of the content and structure of a CLI, initially seemed to be simultaneously a tedious bore (only a bunch of cryptic words, no images?) and a memory challenge (learning how to string those same words together to make software do something?).
However, the usability study of this CLI turned out to be one of the favorite usability studies that I've conducted in the past decade. The fact that those words come out of my mouth still makes people who know me, even a little bit, laugh. What was so great about this study?
What made the study great wasn't just the team's ability to follow through on the findings from the usability study; thankfully, that happens regularly, though to varying degrees. Nor was it the rich feedback that we did indeed receive from the usability participants themselves. What made this usability study great, for me as the researcher, was the commitment of the product team. It's the most dedicated team with which I've ever worked on a usability study.
The software engineers on the product team were committed to hearing what actual breathing users had to say about the proposed changes to the CLI, which is rare, particularly in the context of what was a politically charged project. They hadn't made the changes to the CLI lightly, and they were passionate about making sure that what they had come up with would work for their users. In addition, they were willing to participate fully in the preparation, execution and post-analysis of the usability study, which is a rare occurrence in a field in which usability studies are often used as after-the-fact rubber stamps to mollify potential internal critics rather than to improve products.
Most of the team had never seen a usability study, so we toured the usability labs in Menlo Park, California. After a discussion of various research methods, they accepted that questions about a statistically significant population of users had no place in what we were about to do. Their commitment also involved spending painstaking hours with me, preparing me for the potential questions of live participants, by explaining how the most popular commands were executed both in the original and the proposed CLI, and, most interesting, how it connected to the underlying software structure. They not only attended the usability sessions, but mandated that other engineers, doc writers, and marketing staff on the project attend as well. My manager, who dropped by one of the usability study sessions, said he couldn't enter the observation room (of our largest lab, nonetheless) because it was chock full of observers.
And all this for a usability study for a bunch of words. Just kidding.