Monday Dec 10, 2007

Sun's usability test labs in Menlo Park

Back in August, Jiri Mzourek told us about the building of Sun's usability test labs in Prague, Czech Republic. In October, Kristin Travis told us what it was like to have her engineering team view her usability tests remotely. And in November, I posted an interview with Kim Arrowood, who manages Sun's usability test labs in Menlo Park, California. Now in this post, Kim takes us for a virtual tour of the labs in Menlo Park.

Kim Arrowood has worked in xDesign for over a year managing Sun's usability test labs in the U.S. Before coming to xDesign, she worked at Sun for 6 years in market development engineering as a program manager. Kim is working to improve the visibility of the usability labs in the U.S.


Jen: So Kim, tell me about the usability labs in Menlo Park.

Kim: The labs have both digital and analog recording; we use Camtasia for digital recording, and DVDs for analog recording. We recently installed all new equipment in two of our three labs in Menlo Park, so the labs are really state of the art. We primarily use two of the three labs in Menlo Park and the third lab is used as a staging area for tours and other demo setups. One lab is set up like an office environment, with desks, chairs, and computer equipment. We typically use that for one-on-one (facilitator:participant) usability testing.

The other lab that we use a lot of the time, called the "playspace", is set up in a more creative and casual way. There is a table in the middle with chairs around it, couches, and it's decorated in a more artistic way. It's been built to look more like a design studio than a typical usability lab. For example, it has lamps off to the sides, instead of being lit from the ceiling, and we have toys scattered around the room. We only have one computer set up in the room, and it's off to the side.

Jen: So how do you use the playspace?

Kim: It's great for focus groups, and we record webinars (training) in there. It also has a ceiling-mounted camera that looks down on the table, so we can use it for testing consumer devices or for capturing drawings. Once a week, the playspace is used to host a "design cafe" for teams to strategize and brainstorm, or for people to review their current designs and get feedback on what they are working on.

All of our labs in Menlo Park have an attached control room, separated from the lab by a half-wall and a two-way mirror, but they vary in the lab size and the number or observers they can accommodate in the control room. The playspace can accommodate up to 20 observers, and the other labs can handle up to 10 observers. Each lab also has the ability to support remote observers, for people who can't observe a study in person. This is very useful when part of a team in somewhere else and they can see everything that is going on in our labs.

Jen: So what else should we know about the labs in Menlo Park?

Kim: We've given tours to several different organizations internal and external to Sun. We were part of the CHI 2007 lab tours, and we just gave a tour to the SEED mentoring participants.

Jen: When you give tours, what's the feedback like?

Kim: They think that the control rooms look like a newscast. And the most common question is, "How do you get anything done in the play space?" I tell them that it facilitates creative thinking and communication.


Jen McGinn is an interaction designer in xDesign who is working to improve the user experience with software installation and registration. She has an MS in Human Factors in Information Design and works out of Sun's campus in Massachusetts.

Friday Nov 09, 2007

What is it like to work in a design group, when you're not a designer?

Kim Arrowood has worked in xDesign for over a year managing Sun's usability test labs in the U.S. Before coming to xDesign, she worked at Sun for 6 years in market development engineering as a program manager. Kim is working to improve the visibility of the usability labs in the U.S.


I recently spoke with Kim Arrowood about what it's like to join a design group, when you're not a designer.

Jen: So Kim, tell me a little about what it is that you do.

Kim: I manage our usability test labs. World-wide, we have 9 or 10 labs spread across Prague, Massachusetts, Colorado, and California, but I primarily manage the 3 labs we have in Menlo Park, California. I handle logistics, recruit usability test participants, and help out with technical equipment. I also manage some aspect of operations for our organization, like goals, budgets, and dashboards.

Jen: From your perspective, what's the most challenging or interesting part of coming into a design group?

Kim: The most challenging aspect is the terminology. In my former group, we used the terminology of the customer, but the design group uses both the terminology of the engineering teams as well as terms that are specific to design or usability. For example, I had to learn what it was an interaction designer does and how that is different from the work of a visual designer. And I didn't know what a usability test was until I got to see one, so there was a big learning curve.

One really interesting thing that I learned was how "hands on" design is. I never knew all the work that goes into creating designs before they go to engineering. And I was surprised at how collaborative the design process is. When I worked in engineering, a single person wold work to resolve a single customer problem. But here, there's a very supportive environment -- a lot of teamwork.

Jen: How do you see that manifested?

Kim: Well, when Kristin was working on some designs for the Identity Manager team she took them to the weekly Design Cafe, to get feedback and input on her ideas from other designers in the group. And we have those design cafes weekly, so anyone with an idea or a new mock-up can get feedback from their peers, in a supportive way. But I was surprised, too, at how small the group is, when design is so important to Sun.

Jen: So what is the most interesting part of your job?

Kim: I get to learn a lot more about the products we make; what they are and what they do. I'm reading as much as I can about design and usability testing, but I like to learn about our products by being the participant in our dry runs -- the practice round of a study, when the lab setup and script get tested.

I enjoy participant recruiting, but it's challenging. It's really hard to find good participants; ones that match the test goals for each study.

But the best part of my job is getting involved in the projects, and working on the teams. Everyone works together and communicates -- there are no funny looks and no stupid questions. I really enjoy the collaboration and the teamwork.

Friday Sep 14, 2007

A Sociologist in a Technologist's World: What's a CLI, again?

Nalini Kotamraju is a user researcher in xDesign, and a PhD in Sociology. She has a penchant for research methods and telling it exactly like it is.

Years ago, shortly after I joined the Software User Experience Group (xDesign) at Sun, my manager asked me whether I would be willing and able to conduct a usability study of a new CLI for one of our software products, Sun Cluster. I, the ever eager new employee, promptly responded yes, that I'd be thrilled to do such a study. I then withdrew to my desk, and typed "CLI" in Google to figure out what it meant.

CLI stands, of course, for command line interface, which is a way to interact with software or an operating system. Once I met with the product team and had my first look at the CLI, I understood why my manger had wanted to feel out my reaction to this kind of study. By the time I joined Sun I was a veteran at usability studies, having led many a user through a graphic interface in paper prototypes or interactive mock-ups (usually web sites of now-failed dot.coms). Testing the intuitiveness of the content and structure of a CLI, initially seemed to be simultaneously a tedious bore (only a bunch of cryptic words, no images?) and a memory challenge (learning how to string those same words together to make software do something?).

However, the usability study of this CLI turned out to be one of the favorite usability studies that I've conducted in the past decade. The fact that those words come out of my mouth still makes people who know me, even a little bit, laugh. What was so great about this study?

What made the study great wasn't just the team's ability to follow through on the findings from the usability study; thankfully, that happens regularly, though to varying degrees. Nor was it the rich feedback that we did indeed receive from the usability participants themselves. What made this usability study great, for me as the researcher, was the commitment of the product team. It's the most dedicated team with which I've ever worked on a usability study.

The software engineers on the product team were committed to hearing what actual breathing users had to say about the proposed changes to the CLI, which is rare, particularly in the context of what was a politically charged project. They hadn't made the changes to the CLI lightly, and they were passionate about making sure that what they had come up with would work for their users. In addition, they were willing to participate fully in the preparation, execution and post-analysis of the usability study, which is a rare occurrence in a field in which usability studies are often used as after-the-fact rubber stamps to mollify potential internal critics rather than to improve products.

Most of the team had never seen a usability study, so we toured the usability labs in Menlo Park, California. After a discussion of various research methods, they accepted that questions about a statistically significant population of users had no place in what we were about to do. Their commitment also involved spending painstaking hours with me, preparing me for the potential questions of live participants, by explaining how the most popular commands were executed both in the original and the proposed CLI, and, most interesting, how it connected to the underlying software structure. They not only attended the usability sessions, but mandated that other engineers, doc writers, and marketing staff on the project attend as well. My manager, who dropped by one of the usability study sessions, said he couldn't enter the observation room (of our largest lab, nonetheless) because it was chock full of observers.

And all this for a usability study for a bunch of words. Just kidding.

Wednesday Aug 22, 2007

Building of Two Usability Labs in Prague, Czech Republic (Part 2 of 2)

Jiri Mzourek is a senior manager in xDesign, responsible for Sun Developer Products and SOA/BI. In his spare time, he evangelizes usability in the Czech Republic by organizing SIGCHI meetings, World Usability Day, and working closely with the Czech Technical University on usability and accessibility related projects.

When Sun's Prague office became too small, we all moved to a new building. So during the space planning, we made sure that we got a room there for a usability lab. Why build a second lab? The main reason was to have it in the same place as the engineering team so they could easily attend the test sessions.

This time, we decided to do everything on our own: one of our interaction designers, Rudolf Bock, selected the equipment. Based on our experience with the labs at CTU and at Sun in Silicon Valley, we created blueprints and made sure to have a big one-way mirror. From our experience, despite the fact that some participants feel less comfortable in this set-up, it makes a difference to observers -- they feel more connected to the participants.

We got the space with a one-way mirror in the Summer of 2006! Here are pictures of what we have now:

The lab is fully digital, again partially based on Morae. The equipment is:

  • 2x Dome camera Panasonic WV-CS570
  • 1x Panasonic WV-CU360CJ
  • 1x Data multiplex unit Panasonic WJ-MP204C
  • 1x Blackmagic Design DeckLink Multibridge Extreme PCIe
The first usability study was conducted in this lab in November 2006. Since then, we have used this new lab for the majority of our studies. We also plan to show it to the public as part of World Usability Day 2007. Stay tuned!

Monday Aug 13, 2007

Building of two Usability Labs in Prague, Czech Republic (Part 1)

Jiri Mzourek is a senior manager in xDesign, responsible for Sun Developer Products and SOA/BI. In his spare time, he evangelizes usability in the Czech Republic by organizing SIGCHI meetings, World Usability Day, and working closely with the Czech Technical University on usability and accessibility related projects.

In 2004, our xDesign team in Prague was facing a problem: the number of usability studies we needed to do in Prague kept growing and growing. Since we had no usability lab, we did all of our testing in two meeting rooms that the rest of the company also used. One served as observation room and the second one as testing room. It worked ... but there were two main issues. We had to "build" the lab every time from scratch, and it took about a day to run wires, set up the computer and camera equipment, and move furniture. The second, even bigger, problem was not having enough space. Sun had started its expansion in Prague, so we were hiring a lot of new people, and the building was very crowded. It became harder and harder to find two meeting rooms that were next to each other and available for a couple of days. And it was impossible to build a lab in the building: there was no space for it!

So I started a discussion with the Department of Computer Science at Czech Technical University. Our history of cooperation had started earlier, in 2003, when I was networking with other interaction designers in the Czech Republic. I found out that the person in charge of Czech SIGCHI was my former professor, Prof. Pavel Slavik. So I contacted him, and we quickly found that both Sun and CTU were interested in cooperating in the field of usability. But that's a different story, which deserves its own post.

Then in 2004, we discussed usability labs, reached an agreement and made a deal: Sun would supply the equipment and know-how, and CTU would supply the space and construction. Both institutions would share the facility, and, after three years, CTU would keep all usage rights and equipment. To construct the lab, we worked closely with the manager of Sun's U.S. usability labs at the time, J.O. Bugental, and we outsourced the equipment and configuration work to a vendor.

The lab was designed and built to contain both standard analog technology (a scan converter and DVD burner) as well as fully digital technology, which is currently mostly running on Morae. There is no one-way mirror -- we observe the tests using monitors and video cameras.

The lab officially opened on November 15, 2004, and the Czech Minister of IT, Vladimir Mlynar, attended. It was the first usability lab in the Czech Republic and, most likely, in all of Eastern Europe!

After the lab opening, we also supplied the promised know-how in two ways. First, we arranged for an external company, Relevantive, to provide a four-day training for teachers and Ph.D. students, which covered usability basics including usability evaluation. Second, we cooperated on ongoing projects, coaching and mentoring students as well as teachers.

Since 2005, CTU added usability to its standard curriculum and became the Czech Republic's leading university in this field. Hundreds of students have access to the lab every year and use it to run their accessibility and mobile device projects.

So this is the story of the first lab. I will talk about the second lab in a later post.

Wednesday Aug 01, 2007

Deconstructing the www.netbeans.org Redesign

Jiri Mzourek is a senior manager in xDesign, responsible for Sun Developer Products and SOA/BI. In his spare time, he evangelizes usability in the Czech Republic by organizing SIGCHI meetings, World Usability Day, and working closely with the Czech Technical University on usability and accessibility related projects.

In May of 2006 Jan Rojcek began a redesign of the netbeans.org web site based on the results of some out-of-the-box usability tests that we'd conducted. You can find one example of the test on our opensource website ui.netbeans.org.

The main issues were:
  • The design didn't work well for a new visitor (potential user)
  • To see a NetBeans screenshot, a visitor had to select the correct link from the 42 available links on the home page, and 38 links on NetBeans product page
  • A visitor had the same problem (choosing the right link from 42 or 38) when trying to get to a comprehensive feature description
  • There were 4 pages describing what NetBeans was (First time here?, IDE, Switch, About)
  • Download took at least 3 well-aimed clicks
  • Usability study findings:
    • 3 participants (out of 8) couldn't find how much NetBeans cost!
    • 5 participants reported missing screen shots and too much text on the web site
    • 3 participants reported they had to browse too many pages to find basic information

With this (scary) list of issues, Jano got a "go" to go ahead with the redesign.

He worked with the stakeholders (NetBeans engineering, marketing and webteam) to agree on the main goals of the redesign:

  • New visitor (potential user – not currently using NetBeans) is our primary target
  • Make the basics clear
    • “What is it NetBeans?”
    • “How much does it cost?”
    • “What is so good about it?”
    • “Why should I start to use it?”
  • Make download straightforward
  • Make NetBeans.org more attractive
The redesign was focused on the homepage, product page, download page, docs and support page.

Jano created the first sketch of the new homepage:

He sent it to Leos Tronicek (a visual designer), who created two options:

Stakeholders picked the blue one. So Jano, Josef Holy (another interaction designer) and the NetBeans web team fine tuned that one and created a prototype, which was put on staging server.

So, what was next? Of course, Jano wanted to make sure the redesign met the design goals, so he created a script for a second usability study, which was then conducted in September 2006 by Jakub Franc (a user researcher) and Josef Holy. Sorry, that test report is not public, but here is a list of the main issues found with this design for the homepage:

  • The upper banner with the main download button seemed to be ignored by significant number of participants.
  • A majority of participants complained that information on the homepage did not inform them about NetBeans sufficiently. They expected "short", "summarized", "introductory", "high level" information about the product.

Based on those results, Jano polished the design:


The new website was launched on October 30, 2006, on schedule.

For NetBeans 6.0 there will be couple of new changes, driven mainly by simplified NetBeans packaging and download. We'll get rid of the whole "Add-on" section which will mean updating the layout of the whole front page. Details are still TBD.


About

xDesign is a software user experience design group at Sun.
Follow us on Twitter : Flickr : Blog (see feeds below)

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today