Sun's Recommendations Carousel: Results and Learnings
By Gary Zellerbach on Jan 26, 2010
In September, 2009, we released version 2.0 of the Sun Machine Learning Engine (SMILE), Sun's own advertising-oriented predictive analytics system. The primary customer facing manifestation of v2.0 is the yellow-bordered SMILE Recommendations "Carousel," which you see on many sun.com web pages:
As it's been more than four months since its release, I'll present some of the measured results. Overall, we've been very pleased with the value delivered by the carousel. (There have been some negative reactions, too, but I think that's to be expected when introducing highly-visible new advertising into the mix. More on that later...)
Bottom line, the carousel has exceeded our expectations and far outperformed the small ads we displayed using SMILE v1. Ads (also referred to as "recommendations") in the carousel were clicked by our visitors as often as one million times per week, yielding an average click through rate of over 3.6% (and the rate was much higher even on some sections of the site as opposed to others, depending on the audience). If you're familiar with online display advertising, you'll know that's a pretty phenomenal CTR, and we attribute it to the effectiveness of the SMILE analytics and recommendation engine, strong inventory of content and offers, and highly visible placement.
Clicking on an ad, though, is just the start, so we also measure what happens next, our so-called "key success metrics." These include:
- Downloads Initiated
Many ads point to either the Sun Download Center (encouraging visitors to get our free software) or specific software downloads, and SMILE drives on average 14-16,000 downloads per week. This is very beneficial, as getting our software in customers' hands is often their first real engagement with Sun.
- Offers Obtained
We provide many white papers that require simple registration or login to obtain, and the carousel often advertises and links to these offers (here's an example offer). They provide valuable contacts and leads to our Sales team, and SMILE drives on average 6-7,000 of these new contacts per week.
- TeleWeb Success
Here we measure how many visitors initiate a contact using the "TeleWeb widget" (pictured at right) after clicking a SMILE recommendation. Each such contact has a significant potential sales value (that we've measured accurately over time), so it's significant that SMILE delivers 100 or so contacts each week.
- Sun Startup Essentials (SSE) Applications Submitted
The SSE program is a win-win for Sun and startup companies, providing great value to each over the long run. SSE created a number of SMILE ads, and they perform very well, driving up to 25% of the weekly applications received into the program.
We measure many other variables and events, and in fact, thanks to recent enhancements, can now do closed-loop reporting with some of our Tele-Sales teams. This takes reporting one step further, providing the actual potential dollar value driven by SMILE. Here's how it works: When contacting a lead generated through a SMILE click, sales reps enter into their CRM system an estimate as to the potential value of the lead. This doesn't include all leads generated, as they go to different teams, and not all of them have enabled closed-loop reporting. There is also a time lag between lead generation, dissemination, contact, and possible assignment of marketing pipeline value into the system. Even with those caveats, SMILE generated over $2,500,000 in potential leads value in December, 2009, alone! This is a great example of our push towards measurable, "deterministic" marketing and how we can realistically start to calculate ROI on this project.
Our real-world experience also comes with customer feedback, and we see room for improvement in the following areas:
- First, do a better job of surfacing and explaining how the carousel and the recommendations work. The info has always been there but was "buried" on the All Recommendations Page. We would like to add an "About Recommendations" link to the bottom border of the carousel so that curious and/or concerned (with privacy) visitors can easily learn more about the program, how it works, and how we handle privacy related matters. The link would go to the bottom of the All Recommendations page where we've recently added a new section on recommendations, privacy, and program FAQs.
- When a visitor closes the carousel, it stays closed for the duration of the browser session. But for users who visit often, this was inadequate. We would like to implement a different solution that keeps the carousel closed longer for users who don't wish to view it.
- Due to an initial basic mapping system between images, products, and ads, we sometimes show duplicate images and/or lack image variety. We've been adding new images to the system to address this.
- There are a few more simple enhancements that will also increase variety in the carousel, such as not showing slight ad variations for the same product or offer at the same time, and eliminating ads that link to the page the user is already on.
- Lastly, on the back-end, we continue to refine and enhance the recommendation engine and methodologies with a goal of always increasing the relevancy of the ads to our visitors.
Will these enhancements see the light of day? Some decisions are pending the closure of the Oracle acquisition, so time will tell.
Regardless of what happens, though, I hope this information helps convey the strong results the program has produced, the success of our predictive technology, how it can be further improved, and the promise such systems hold for the future.