Integrating A/B and multivariate tests with a third-party analytics platform is common practice. With these integrations, you can see data in your testing platform about the population of users that visited each experience of your web or mobile tests. In addition, you can separately analyze and compare each group of users that saw each individual experience using the metrics provided by the analytics platform. The extra insight gained from this new data is valuable.
Therefore, the testing platform and the third-party platform provide two separate ways to analyze the same population. An analyst would expect the two population counts to be roughly equivalent. However, there are sometimes differences between the population counts in the two platforms for various reasons.
To avoid data discrepancy, pay close attention to these three factors when integrating data from a testing platform and a third-party platform:
Difference in Definition
The largest source of population count discrepancy is a difference in how “population count” is defined, especially around selecting specific date ranges to view. For example, the definition of the testing platform’s population count might differ from that of the third-party analytics platform: The testing platform might only count new visits during the selected date range, while the analytics platform might count all visits in the selected date range.
In other words, users who previously visited the test before the selected date range starts and then came back to the test page during the selected date range would not be counted on the testing platform but would indeed be counted in the analytics platform. This would lead to a higher count on the analytics platform.
The testing platform and the analytics platform may also be filtering data by different browser rules. All analytics platforms let you customize various settings for filtering out unwanted data, and these settings are often different from those of the testing platform.
If the testing platform is allowing all users from any desktop or tablet browser into the population, but the analytics platform is counting desktop, tablet, and mobile from only the latest versions of the four ‘power browsers,’ then two entirely different sets of users would result. (The four ‘power browsers’ refers to the four most popular browsers: Chrome, Firefox, IE, and Safari.)
Time Zone Cutoff
When filtering data by a specific date range in a testing platform and analytics platform, the results on each platform could be slightly different due to time zones differences, even if that date range is the same for both.
At what time do the two platforms cut off the data at the start and end of your selected days? Midnight UTC Monday is 8 p.m. EST Sunday, for example. If these cutoffs are different between the two platforms, then there will be a few hours of difference in what range is being viewed. And if those few hours occur during a high-traffic period, then the difference in population count could be significant.
Enriching your testing data with a third-party data platform will enhance your insight into customer behavior. As long as you pay close attention to the factors that can cause discrepancies between the two data sets, this additional data can be accurately analyzed and compared.