The difference between one-tailed and two-tailed testing

Why should you do A/B and multivariate tests?

At eTail West, one wise retailer told the audience, “80% of what you think you know about your site is wrong.” His message was one among many ways digital marketers urged their audiences to test (and test often).

They said this back in 2015, but it still holds true today.

The problem is, marketers often assume testing in itself is enough to negate the impact of personal bias, even as many brazenly ignore the statistics behind running a valid test. The results are several surprisingly common misconceptions that can turn even the most well-intentioned tester into a mindless, hypothesis-confirming drone.

One of the biggest mistakes a marketer can make is failing to understand the difference between one-tailed and two-tailed tests. And we don’t blame them. Testing vendors don’t necessarily provide the option to calculate statistical significance in more than one way, and if they don’t, they probably aren’t going to bother explaining the difference.

I’m not here to say a one-tailed test is inherently useless, but rather it's a risky point of confusion when understanding the validity of your testing campaigns and making decisions about the user experience on your site or mobile app.

Need help with testing? Check these guidelines for best results when testing with Oracle Maxymiser Testing and Optimization.

Now let’s get into some detail.

One-tailed tests: What is a one-tailed test?

A one-tailed test allows you to determine if one mean is greater or less than another mean, but not both. A direction must be chosen prior to testing.

In other words, a one-tailed test tells you the effect of a change in one direction and not the other. Think of it this way. If you're trying to decide if you should buy a brand name product or a generic product at your local drugstore, a one-tailed test of the effectiveness of the product would only tell you if the generic product worked better than the brand name. You would have no insight into whether the product was equivalent or worse.

Since the generic product is cheaper, you could see what looks like a minimal impact but is, in fact, a negative impact (meaning it doesn’t work very well at all!). But you go ahead and purchase the generic product because it's cheaper.

If this is the case, you’re probably wondering when a one-tailed test should be used. One-tailed tests should be used only when you are not worried about missing an effect in the untested direction.

But how does this impact optimization? If you’re running a test and only using a one-tailed test, you will only see significance if your new variant outperforms the default. There are two outcomes: 1) the new variants wins or 2) we can't distinguish it from the default.

Here’s a quick summary:

Find out how National Express, the UK transport brand, improved its website engagement metrics by 22% with Oracle Maxymiser Testing and Optimization.

Two-tailed tests: what is a two-tailed test?

A two-tailed test allows you to determine if two means are different from one another. A direction does not have to be specified prior to testing. In other words, a two-tailed test will take into account the possibility of both a positive and a negative effect.

Let’s head back to the drug store.

If you were doing a two-tailed test of the generic against the brand name product, you would have insight into whether the effectiveness of the product was equivalent or worse than the brand name product. In this instance, you can make a more educated decision. If the generic product is equivalent, you would purchase it because it's cheaper, but if it's far less effective than the brand name product, you’d probably shell out the extra money. You wouldn’t want to waste your money on an ineffective product, would you?

So when should a two-tailed test be used? Two-tailed tests should be used when you're willing to accept any of the following: one mean being greater, lower, or similar to the other.

And how does this impact optimization? When running a test, if you're using a two-tailed test, you'll see significance if your new variant’s mean is different from that of the default. There are three outcomes: 1) the new variant wins, 2) the new variant loses or, 3) the new variant is similar to the default.

Here’s a quick summary:

Which testing approach should you be using?

Two-tailed!

Two-tailed tests mitigate the risk involved with predicting how future visitors will be impacted by the tested content. By accounting for all possible outcomes, this approach provides more valuable, unbiased insights that can be reported on with confidence.

Testing is supposed to make it easier for marketers to understand the impact of a certain change without IT intervention, but when the difference between one-tailed and two-tailed tests goes ignored, both the marketer’s time and IT resources are at risk of being wasted. Don’t fall victim to personal bias or winning results which are vacant of real meaning. This testing approach does require more traffic and time, but that's a small price to pay for reliable results.

Now get testing!

Additional Resources

• Visit the Help Center for Oracle Maxymiser Testing and Optimization to get guidelines on testing for best results.
• Find out more about A/B Testing with Oracle Maxymiser Testing and Optimization.
• Find out how National Express, the UK transport brand, improved its website engagement metrics by 22% with Oracle Maxymiser Testing and Optimization.

Make the best use of data in your marketing campaigns for B2B and B2C.

• See how Oracle B2C Marketing can help you capture real-time customer behaviors to create more effective, data-driven marketing campaigns.
• See how Oracle B2B Marketing can help you use real-time data insights to engage, convert, and nurture leads.