When we talk about tests "failing," we’re not talking about if the test produced winning experiences or losing experiences. In fact, if a test offers clear insights, whether those be positive or negative, it's a success! Tests "fail" when they lack a clear hypothesis, are poorly structured, and are stopped before reaching statistical significance.
So, how can you avoid these three common mistakes? Let's go into a bit more detail...
1. Feckless Foundation
One of the most critical steps in testing arrives at the outset. The moment a valid test idea is conceived, you must create a corresponding hypothesis. This hypothesis or hypotheses will often determine the success or failure of a test. Without it, test building and analysis will fall far short of productive.
Develop your hypothesis or hypotheses around the foundational question: “What is the primary goal(s) of this test?”
Clearly determining and stating the goal of your test will guide you throughout the process. For example, hypothesizing about test objectives will help you determine what key performance indicators should be tracked. A refined set of appropriate KPIs is vital to the success of your test. Without predetermined KPIs you’ll be left frustrated at the test analysis stage, lacking the tracking you need to properly move forward. The results are untimely campaign updates, delayed results, and unhappy stakeholders.
2. Shaky Scaffolding
This is another early stage in the campaign lifestyle that can undermine the success of a test. Figuring out how a test will be structured is an essential piece of the puzzle. You don’t want to find yourself having built out a complex multivariate test only to discover you don’t have the traffic you need to support a test of that size.
This why when planning a test, the amount of traffic to the test page and the number of primary KPI actions that occur need to be carefully accounted for. We recommend using our Test Duration Calculator to help you determine the number of experiences your test page can support. When you have this information you can decide if your test needs to be restricted to A/B or if it can be a complex multivariate.
3. Impatience, Impatience, Impatience
Failing to let a test reach maturity, and therefore statistical significance, can be the final nail in your optimization results coffin. (Learn more about this topic on our post "Statistical Significance 101!") If you have avoided the first two common mistakes by hypothesizing thoroughly and planning your test structure properly, then you have set yourself up for success. By this point, you should also have a rough idea of how long you will need to let your test run in order to reach statistical significance.
Let’s say the Test Duration Calculator shows you will need 3 weeks to reach statistical significance, but that after 5 days you see a 100% Confidence Index. Is it time to have a conversation about whether or not you can declare a winner? Yes. Will you like the answer? Probably not, because the answer is almost always no.
Failing to let a test reach maturity, and therefore statistical significance, can be the final nail in your optimization results coffin. You should have a rough idea of how long you'll need to let your test run.
Most likely, we’d conclude the test needs to reach maturity because we have not yet collected enough data to reach ‘true’ statistical significance. At that point you should go back, consider the actual number of generations, primary KPI actions, and number of experiences, and use the Test Duration Calculator to see if the amount of time needed has been met. If not, continue to run the test as planned.
If you follow our steps to avoid the most common mistakes when it comes to setting up, running, and concluding a successful test, you can come out of each test with rich and statistically significant insights to bring to your stakeholders. You might even want to make yourself a testing checklist or pin this post to your wall to remind yourself to avoid these costly mistakes! Here’s a useful resource for test setup that will keep you focused on the fundamentals next time you run a test.