Predictive Analytics: Measuring the Results
By Gary Zellerbach on Aug 28, 2009
I wrote an overview earlier this month about Predictive Analytics (PA), what it is and what sorts of benefits it offers. I've also written about the Sun Machine Learning Engine (SMILE) Project, putting PA to work on Sun's web sites. Today marks the end of SMILE Phase 1, as we are in the midst of releasing SMILE v2.0. You'll see the results on www.sun.com over the course of the next month, as we gradually enable more and more of the site with our new "Recommendations for you" carousel. Here's what you'll be seeing soon!
So I thought today would be a good time to touch on the results from SMILE 1.0. The initial release was simply about serving small text-only ads in the right hand column on many sun.com sites. Here's an example of a SMILE-served ad (outlined in red):
To evaluate results, we calculated standard metrics, such as impressions (number of times the ad was displayed), clicks (number of times the "call to action" link in the ad was clicked), and CTR (click-through rate -- number of clicks divided by number of impressions, as a percentage). I'm not able to share the detailed CTRs at this time, but suffice it to say they're low. That's not unexpected -- these are small ads, easily overlooked or ignored, and you shouldn't expect a lot of clicks on this type of banner ad.
However, we also calculated Uplift, and that's where we looked to measure the power of our analytics. We did not have recommended ads for all customers, just return visitors with anonymous cookies that we recognized from earlier visits. Thus, we served many default ads, as well as many recommended ads. By comparing the CTRs of both, we can explicitly measure the influence of our predictive system, measured as "uplift." Here's a simple example:
- On a web web page, we show 100 SMILE-recommended ads that get 15 clicks, for a 15% CTR.
- On the same page, we show 100 default ads (same size, same location, just not personally targeted), and they get 10 clicks, for a 10% CTR.
- The SMILE Uplift in this case is 50% ((15-10)/10 \* 100).
We carefully tracked SMILE Uplift for the last five months, and we saw an average uplift of 58.3%. As we serve millions of ad impressions, that translates into 1000's of additional clicks generated by our PA system. The ads often point to downloads or white paper offers that customers sign in to get, and thus we collect 1000's more contacts and what they're interested in, which we can then (hopefully) turn into qualified leads and ultimately new customers. So we can see a definite ROI for this effort. And keep in mind this was "version 1" of the analytics, which we're continuously refining, enhancing, and developing -- we expect ongoing improvements in future results.
Actual weekly Uplift gyrated pretty wildly -- here's a summary chart:
You can see general improvement over time as we improved the algorithms, steadying for the most part in the 40-80% range. In the last week, we released SMILE ads on the Sun Download Center, which had (as you can see) an interesting impact on Uplift! SDLC gets a huge volume of visitors, and most users are there to download and nothing else. We also found a large proportion of users there for whom we did not have recommendations (either because they were new or they'd deleted their Sun cookies). The result was a pretty big dip in CTR for the default ads, while we held steady on the recommended ads, thus the skyrocketing Uplift score the last week.
With the release of SMILE 2.0, we're completely changing how we do our measurements (it's a long story), so we'll be tweaking our weekly measurement system and reporting. We'll have new functionality and new measuring capabilities, and I'm looking forward to seeing the results from our newest release.
As I hope these numbers portray, we've demonstrated solid benefit to our emerging PA technology. It's a great start, but there's still a lot of upside potential remaining -- we're optimistic of delivering even more dramatic results in the future.