In the first blog post of this series, we focused on introducing usability testing and the importance of establishing a study protocol, followed by recruiting, and discovery. Now it’s on to analyzing the results!
The team is exhilarated as the final usability test comes to an end. There’s a massive treasure-trove of information before us, and we can’t wait to start filtering through it. Right away, some folks are excited to have found exactly what they set out to see. Others are thrilled to have unearthed nuggets they never expected to turn up. Everyone finds value in the experience, but it’s important to take a step back and make sure no stone is left unturned. This is where analyzing the data comes in.
On average, every usability test has four to eight participants with multiple team members sitting in to hear what they have to say. This means there’s a great deal of footage and a stockpile of notes to sift through. For me, creating a spreadsheet has always been the best approach. There’s a row for every task, and a column for every participant, making the process of transcribing the notes, and then analyzing them, easy.
Sometimes, as I’m copying the notes over to my spreadsheet, I catch a discrepancy. This isn’t surprising given the notes are coming from different sources and perspectives, including myself, my teammates, and our debriefs. When this happens, the videos come in handy to quickly resolve the gaps. I have also found it’s worth going the extra mile and watching the videos from start to finish if time permits. It gives me an opportunity to experience the usability tests from a different vantage point. This time as an observer rather than a moderator. This is a good way to unearth some quotable gems as well.
After transcription, it’s time to start my analysis. First, I read through each task, and look for the number of participants who:
Then, I zero in on each of the issues and tally up the number of participants who experienced them. This process brings to light what we need to fix first. Meanwhile, it’s also important to spotlight where the participants hit the mark, so the team can rope off these areas of the product from further change.
Getting the outcomes to the team is always my top priority. An email with an executive summary, or even a list of bullet points, gets the job done. But more often than not, I like to take things a step further and put together a presentation as an added bonus. This presentation gives me a chance to paint the big picture and share the story of our journey with the rest of the organization, so they, too, can get their hands on this wealth of information.
This blog is the fourth in a five-part series focusing on the journey UX designers must take with organizations as they work with stakeholders to ensure intuitive application interfaces and workflows are designed and implemented.