Thoughts from a recent remote usability study
By Kristin Travis on Oct 25, 2007
Kristin Travis has been working in high tech as an interaction designer and usability engineer for more than 15 years. She is part of the xDesign team based in Menlo Park, California, and she currently supports the Identity Manager team, which is based in Austin, Texas.
The last release of Sun's Identity Manager software (in May of 2007) had substantial user interface changes, so when I joined the team in June we discussed conducting a usability study in the Menlo Park usability labs, to get feedback from representative users on the current release.
In my experience, most development team members appreciate seeing how users interact with a piece of hardware or software that they've helped to create. Seeing first-hand reactions to existing functionality helps to shape team members' thinking about changes and new features for a product.
But while I'm located in Menlo Park, the Identity Manager development team is located in Austin, Texas. So the questions I had going into this exercise were: would it be relatively easy to involve a remote development team in a usability study? And would the remote team be satisfied with viewing a study in real-time, but not actually being in the same room as the user?
So what did we do?
In terms of the setup, we created a VNC connection between the usability lab in California (where I was, with the study participants) to a conference room in Texas, where the members of the development team could observe the test sessions.
The remote access allowed the people in Texas (and other locations, if needed) to see what study participants were doing. The Texas team could see the participant's computer monitor and watch, in real time, while the participant interacted with the product. In addition, the team could listen to the participant over an audio conference call that we established between the locations. At the end of each session, if the remote team wanted to follow up with the participant about a particular issue or question, they could do so by using the conference call.
And how did it work out?Here are some highlights of the feedback that I got from the remote and local teams:
- As with any other type of study, it's really important to conduct a dry run of the session. You don't want to get side-tracked during the study by unanticipated logistical issues. During our dry run, we diagnosed an error in the VNC login instructions for the remote set up. That took a while to figure out, but then things went according to plan.
- The remote team's commitment to the study is essential. Jeff, my main contact in Texas, coordinated the remote conference room, kept everyone there informed about any schedule changes, and attended each study session. Considering the two-hour time zone difference, this meant a few late nights in Austin. But it was extremely helpful to have Jeff complete intra-task dependencies so I could concentrate on working with the participant.
- Jeff said that seeing how the participant interacted with the product was a huge benefit. This was true even though they couldn't "see" the participant directly (as they would have if they had been here locally).
Would we change anything for the next time?
I was in the room with the participant, and Kim, our Usability Lab Manager, was in the control room interacting with Austin by phone, so there was a bit of a communication delay at times. If Kim and Jeff had relevant content to share, Kim had to wait for an appropriate time to break into the conversation that I was having with the participant. It would have been useful to have a '3-way IM chat' up and running, so if the participant discovered any software surprises, or they had any related questions, we could communicate more quickly, without disrupting the flow of the test.
So the questions I had going into this exercise were: would it be relatively easy to involve a remote development team in a usability study? And would the remote team be satisfied with viewing a study in real-time, but not actually being in the same room as the user?
Well, in this case, yes.