While many marketers are adopting A/B and multivariate testing as primary optimization methods, one of the biggest challenges they face is determining if a test was a success. Success doesn’t necessarily correspond to clear and instant uplift on a certain metric; there’s more to it than that.
To come to an educated conclusion about the outcome of your test, there are two major questions you should ask yourself.
1. How Does the Data Look for My Primary Metric?
Before launching a testing campaign, it is necessary to identify a primary metric that will be directly impacted by the changes implemented on the test page. This will help keep the purpose of the test in focus and give you a clear starting point for your analysis of the results.
The first data point you will want to look at is sample size to confirm that each experience in your test had enough generations to return statistically viable conclusions. Next, you will want to look at conversion rates for the primary metric to determine if your variant experiences performed noticeably better or worse than the default.
In conjunction with the conversion rate, it is important to also consider the level of statistical significance. You may have a case where the conversion rate for a variant is higher than that of the default, but if the results are not statistically significant, the difference in conversion you are seeing is most likely due to random statistical fluctuation and not the changes implemented on the page.
If you have a higher conversion rate in a variant paired with statistical significance, this means that the difference in conversion rate can be attributed to the changes made on the page as part of the test and you have yourself a winner.
This same process can be followed for secondary metrics. However, secondary metrics should be viewed as part of helping you build a full narrative; on their own they shouldn't be used as a determination of success or failure.
2. Did I Learn Something That Can Help Influence Future Initiatives?
Testing and collecting data is great, but if you cannot gain any meaningful, actionable insights, what’s the point? If you analyze the data for your primary metric and determine that one of the variants is a winner, you should be able to use those results to either identify another testing opportunity or to implement changes for other places on the site.
For example, if you run a layout test on one product page and find a clear variant winner, you would probably want to implement that same layout across the remainder of product pages, since it has been proven to work.
However, a common misconception is that a test is only successful and actionable if one of the variants wins. In an ideal world, there would be a variant winner for every test. But in reality, tests that have minimal or negative impact can produce equally profound learnings.
Here’s another example to further clarify.
You are unhappy with the performance of a landing page so you choose to test new copy. After the test has reached a conclusion, it is clear that both versions of copy were equally effective. While there is no winner in this campaign, you have learned that copy is probably not the source of the page’s underwhelming performance. As a followup, you may test the imagery instead to see if that is a driver of user behavior.
A successful test can be defined in a number of ways. Yes, the clearest version of success is a winning variant that reaches statistical significance for the primary metric. But if there are clear and actionable learnings to be pulled from the data, whether that data is positive or negative, you have still gained knowledge about your site and now know more about it than before you ran the test.
And that should be considered a success in itself.