Do you ever start out with a simple A/B or multivariate test idea and end up with a complex or even confounding campaign?
It happens all the time. Simple campaign ideas which are intended to answer one critical business question transform into complicated tests after additional parties enter the discussion. It’s not just testing. Even the clearest, most indisputable business goals can become diluted as they make their way through an organization’s various teams. Returning to your original objective is the only way to reorient a project, or in this case a testing campaign and set it on the road to success.
When it comes to testing, extraneous complexity can hamper every detail, from campaign set-up to reporting and analysis.
Here’s an inside look at what can come of an overcomplicated A/B or multivariate test:
Adding peripheral elements to campaigns can lead to UX changes that skew the data for the main goal of the test
Including extraneous metrics to obtain secondary answers can mean having to 'weigh' the impact of the test to determine a winner
Increased test complexity and hours needed for development, QA, and analysis
Having to run follow-up tests to answer the same business questions because no clear conclusion was reached
Here’s what it boils down to: Ideally, the goal of each A/B or MVT should be to answer a single business question.
A simple, well thought out test idea is easily derailed by enthusiastic brainstorming sessions about test concepts. While I know first hand how heated testing discussions can be, we all need to be cautious not to leave these conversations concocting ways to answer five new business questions with the same campaign.
However, sometimes a higher priority business question that can be answered through testing comes up in the process. And that’s great! Now it is time to reevaluate and reprioritize your upcoming campaigns to accommodate this new idea, just don’t haphazardly attempt to add in campaign elements and logic to try to answer an extraneous question.
Sometimes you will be able to re-scope the campaign to incorporate new elements and kill two birds with one stone, but don't lose sight of your goal or inadvertently muddy your results in the process. Take a step back to understand and define the true objective of the campaign. If the campaign is more complicated than it needs to be to answer your question, pare it down. Remember, we can always run follow-up iterations of campaigns based on your original learnings and/or to draw further conclusions.
Paring down campaigns to focus on specific business questions has a number of benefits:
Conclude campaigns with clear results and direction
Decrease test complexity and hours usage
Launch the campaign sooner
Empowers you to move on!
It’s easy to give into the temptation of overcomplicating a test under the perception of efficiency. It’s much harder to define specific objectives and prioritize a business question. With careful planning your testing campaigns can deliver clear results and positive outcomes. Be patient and your outcomes will be clear and indisputable so you can move on to your next big idea knowing that you’ve adequately addressed the previous one.
For more about approaching your next testing campaign, check out our new infographic on One-Tailed vs. Two-Tailed tests.