This week’s guest blog post is contributed by Jonathan Seidner, Senior Director, Engineering, Oracle Data Cloud.
Measuring the performance of cross-device data might seem simple at first, but unfortunately, this is not the case.
In an earlier post, we discussed the importance of understanding the inherent trade-off between precision and recall in cross-device mapping.
This post will address some important pitfalls involved with using truth sets to evaluate and compare cross-device maps from different vendors.
A truth set is a list of device pairs (device IDs and/or cookie IDs) that are known to be used by the same individual, typically based on identical login data used across different devices.
Each pair in the truth set is identified by a unique user ID. Different pairs may be associated with the same user ID, if more than two devices are determined to be in use by a single user.
The truth set is used to evaluate the performance of a probabilistic device map by measuring how many “true matches” are represented within the device map.
It is important to make sure that the truth set contains each matched pair only once. While this sounds obvious, it is actually quite common to find truth sets that contain the same match listed twice, in different orders.
For example, both PC123-iOS456 and iOS456-PC123 could be listed. Duplicated matches will result in skewed measurement results and must be avoided.
To help prevent this problem, it is strongly recommended that the matches listed in truth sets always follow a consistent order.
For example, matches containing a PC always have the PC ID first, or the higher ID number always appears first. It doesn’t matter what order is selected, as long as it is consistent throughout each truth set, and across truth sets.
This same issue applies, of course, to the device map itself. If you discover that a device map provided by a vendor contains duplicate match listings or inconsistent pair ordering, that should raise a big red flag for you.
Measurements of precision are heavily influenced by the size of the truth set. For this reason, using a small truth set to compare two device maps can generate results that significantly understate differences between the two maps.
A common step in probabilistically matching devices is “clustering” devices according to certain criteria, which is roughly equivalent to only considering devices from the same household.
In other words, only devices belonging to the same household will even be considered as potential matches.
The likelihood of having truth data for more than one person in a household decreases rapidly as the size of the truth data decreases.
This likelihood is directly correlated with the likelihood of incurring a false positive. This is because, in order to incur a false positive, one must incorrectly match two devices that are found in the truth set.
If matching is only done within a household and the truth set is small, then the likelihood for an incorrect match to be considered as a false positive is also small.
To clarify, this means that the likelihood of incorrectly counting a match as a false positive (if it is, in fact, false) decreases exponentially as the size of the truth set decreases.
On the other hand, the likelihood to correctly not count a match as a true positive only decreases linearly as the size of the truth set decreases.
The result of this, as can be seen in the following chart, is that when the truth set is small, the measured precision will be artificially high.
Conversely, the bigger the truth set is, the more accurately the precision will be measured (for a given recall value). The difference is huge.
The conclusion from this is that if one is comparing two device maps using a very small truth set, one device map could be measured with a precision of 97 percent and the other at 97.1 percent, making them seem almost identical when, in actuality, one could be 90 percent and the other 60 percent.
Continue reading in part two, where we take a close look at the dangers of using biased or “dirty” truth sets.
Try the Oracle Crosswise complete device map for one month, updated weekly, in your own systems. Sign up to get a free month of Oracle Crosswise.
Stay up to date with all the latest in data-driven news by following @OracleDataCloud on Twitter and Facebook! Need data-related answers for your next marketing campaign or client partner? Contact The Data Hotline today. (What's The Data Hotline?)
Image: ESB Basic/Shutterstock