You’ve set up and launched your test—but not every test can run from start to finish untouched. Fairly often, there’s a justifiable business reason to change the variant weights of an A/B or multivariate test in the middle of its planned runtime. And that’s OK! I’ll show you how to analyze the data for a test in which you must adjust variant weightings mid-test in a way that doesn’t mislead your company, marketing team, or analysts.
What Are Variant Weightings?
First, let’s define what these are. Variant weights refer to the traffic ratio you split between the default and challenger versions of a test. If you’re running an A/B test, for example, and you decide to direct 23% of traffic to the default page and 77% to the alternate, then the variant weighting is 77-23. This also works if you’re running a multivariate test: If you’re analyzing four total experiences, the variant weights could be 25-25-25-25, or 10-15-30-45.
When Would You Change Variant Weightings?
Many situations can lead to your needing to adjust how your variants are weighted.
A common instance is if your alternate experience is converting much higher than the default during the test; this could encourage you to drive more traffic to it than you were originally. This is called throttling—increasing a variant’s weight because it’s doing so well you want to explore its uplift potential.
Another opportunity is the opposite of this: Stakeholders can ask for a variant to be removed from a test if it’s sharply underperforming. This way, traffic can be directed to better-performing variants that yield higher overall conversion rates.
Let’s Get Specific
Take a look at the A/A test below. (An A/A test validates data by testing it against itself; you can learn more about them in our previous post here.) We started Week 1 with a 90/10 split in traffic and in Week 2 changed that to a 50/50 split. Assuming the conversion rate variants will perform the same in the future as they do now, changing variant weights won’t create any issues (E10 and E11 in the “Steady Conversion Rate” table).
However, if we increase traffic this can appear to increase conversion rates (I7 and I8 in the “Increased Conversion Rate” table), causing trouble in data interpretation. Even though we know the variants always perform equally, it looks as if one does better than the other. The variant with increased traffic shows an inflated conversion rate, while the variant with a decreased weighting percentage shows a deflated conversion rate.
Getting Around Deceptive Data
One way around this problem is to view the two date ranges separately: start of test to date of weight change, and date of weight change to end of test. Another way is to view normalized data. Normalized data are results in which each variant has been mathematically adjusted to enable a fair comparison between them, even if variant weights were changed in the selected timeframe. If you don’t normalize your data, it’s not possible to fairly compare your results if you change variant weightings in the middle of a test!
With careful consideration, changing your variant weights mid-test doesn’t have to skew or invalidate your results. Understand when and why you might change them and avoid misinterpreting the data after you do, and you can rest easy next time stakeholders ask for adjustment during a test: You’ll still have valuable data to work with.
Our reporting suite offers a Normalized Conversion and Revenue Report for all campaigns. Find out more in our Support Center.