Welcome to the Oracle Modern Marketing Blog:
The latest in marketing strategy, technology, and innovation.

How to Improve the Quality of Your Optimization Test Results

Harrison Small
Principal Consultant for the Oracle Maxymiser Professional Services team

Many digital marketers who use A/B and multivariate testing as the cornerstone of their optimizations strategy barely scratch the surface when it comes to the data they're capable of collecting. The strategies offered below will enhance the quality of your metrics, your learnings, and therefore the possible positive outcomes of any test.

Here are three ways to improve the quality of your test results:

1. Identify a Sensible Primary Metric

It is integral to identify a primary metric prior to starting a test. The metric will serve as the key piece for determining if the test is successful. The chosen metric should be directly impacted by the changes being implemented on the page as part of the test. To meet these criteria, ensure that the action is in close ‘proximity’ to the test page.

For example, if you are changing content on the first page of a four-page credit card application, it can be tempting to identify the primary metric as completed applications. However, it is more logical that the content on page one would have a bigger impact on whether or not a user moves on to page two, not on whether they complete the application as a whole.

There are a number of non-test related reasons why a user might drop out of the application between pages two and four. Therefore, using application completions as the primary indicator of success can be misleading, while measuring the number of users who reach page two is more directly related to the test being run on page one.

Overall, identifying a sensible primary metric will help you keep the data focused and organized.

2. Utilize Action Attributes Where Possible

Action attributes allow for an increased level of granularity when analyzing test data. Hypothetically, you might have a page with multiple products listed with “Buy Now” buttons next to each product. You have three different options in this scenario:

  1. You can create one click action for all of the “Buy Now” buttons. This would fire on any “Buy Now” click regardless of which product the user clicks and there is no way to tell if one product drove more engagement than another.
  2. You can create a separate “Buy Now” click action for each individual product. This makes analysis possible on the product level, but it also requires much more effort to make sense of the data due to the increased number of metrics.
  3. You can create one click action for all of the “Buy Now” buttons, but also include an action attribute which would associate each click with the relevant product.

Option C is ideal because having one action will keep all of the data manageable, while the attribution allows you to filter campaign data by product to see the number of clicks associated with each product.

3. Segmentation & Personalization Criteria

While action attributes make granular filtering possible on the metric level, segmentation and personalization attributes make it possible to filter on the user level.

For example, if you are curious to see if users who arrive to the page via different channels behave differently, you can create custom segments for users who fall into each traffic source and then look at the data exclusively for users who fall into that segment. Segments can be created based on any identifiable user attribute whether it be age, device, geo-location, etc. These capabilities are often underutilized, leaving valuable, high-quality data untapped.

In the end, there are many steps a user can take to ensure the quality of their test results. Today we’ve covered three ways we ensure that the results of our clients' optimization tests are as good as possible. A lot goes into making a test successful, from set up to metrics, so ensuring you have best practices in place before embarking on any test is vital.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.