Friday Aug 28, 2009

Predictive Analytics: Measuring the Results

I wrote an overview earlier this month about Predictive Analytics (PA), what it is and what sorts of benefits it offers. I've also written about the Sun Machine Learning Engine (SMILE) Project, putting PA to work on Sun's web sites. Today marks the end of SMILE Phase 1, as we are in the midst of releasing SMILE v2.0. You'll see the results on over the course of the next month, as we gradually enable more and more of the site with our new "Recommendations for you" carousel. Here's what you'll be seeing soon!

SMILE Recommendations Carousel

So I thought today would be a good time to touch on the results from SMILE 1.0. The initial release was simply about serving small text-only ads in the right hand column on many sites. Here's an example of a SMILE-served ad (outlined in red):


To evaluate results, we calculated standard metrics, such as impressions (number of times the ad was displayed), clicks (number of times the "call to action" link in the ad was clicked), and CTR (click-through rate -- number of clicks divided by number of impressions, as a percentage). I'm not able to share the detailed CTRs at this time, but suffice it to say they're low. That's not unexpected -- these are small ads, easily overlooked or ignored, and you shouldn't expect a lot of clicks on this type of banner ad. 

However, we also calculated Uplift, and that's where we looked to measure the power of our analytics. We did not have recommended ads for all customers, just return visitors with anonymous cookies that we recognized from earlier visits. Thus, we served many default ads, as well as many recommended ads. By comparing the CTRs of both, we can explicitly measure the influence of our predictive system, measured as "uplift." Here's a simple example:

  • On a web web page, we show 100 SMILE-recommended ads that get 15 clicks, for a 15% CTR.
  • On the same page, we show 100 default ads (same size, same location, just not personally targeted), and they get 10 clicks, for a 10% CTR.
  • The SMILE Uplift in this case is 50% ((15-10)/10 \* 100).

We carefully tracked SMILE Uplift for the last five months, and we saw an average uplift of 58.3%. As we serve millions of ad impressions, that translates into 1000's of additional clicks generated by our PA system. The ads often point to downloads or white paper offers that customers sign in to get, and thus we collect 1000's more contacts and what they're interested in, which we can then (hopefully) turn into qualified leads and ultimately new customers. So we can see a definite ROI for this effort. And keep in mind this was "version 1" of the analytics, which we're continuously refining, enhancing, and developing -- we expect ongoing improvements in future results.

Actual weekly Uplift gyrated pretty wildly -- here's a summary chart:

SMILE Uplift chart

You can see general improvement over time as we improved the algorithms, steadying for the most part in the 40-80% range. In the last week, we released SMILE ads on the Sun Download Center, which had (as you can see) an interesting impact on Uplift! SDLC gets a huge volume of visitors, and most users are there to download and nothing else. We also found a large proportion of users there for whom we did not have recommendations (either because they were new or they'd deleted their Sun cookies). The result was a pretty big dip in CTR for the default ads, while we held steady on the recommended ads, thus the skyrocketing Uplift score the last week.

With the release of SMILE 2.0, we're completely changing how we do our measurements (it's a long story), so we'll be tweaking our weekly measurement system and reporting. We'll have new functionality and new measuring capabilities, and I'm looking forward to seeing the results from our newest release.

As I hope these numbers portray, we've demonstrated solid benefit to our emerging PA technology. It's a great start, but there's still a lot of upside potential remaining -- we're optimistic of delivering even more dramatic results in the future. 

<script type="text/javascript"> tweetmeme_url = ''; tweetmeme_source = 'garyzel'; </script> <script type="text/javascript" src=""> </script>

Thursday Aug 06, 2009

If You Predict It, You Own It

As I've written about previously, I'm currently the Project Manager for the Sun Machine Learning Engine (SMILE) project, based on predictive analytics (PA) technology we're developing in-house. While I have a lot of experience building and managing complex web systems such as this, I haven't worked with PA technology before. I set out to learn more about it, for two reasons:

  1. Since I'm the PM for this project, it's generally a good idea to know what I'm doing and talking about!
  2. ROI is very important, both for this project and for ensuring the ongoing application of PA technology in general at Sun. This matters, of course, to Sun's management, and as you might imagine, it can't hurt to convey these benefits to our soon-to-be new owners as well. 

So, I set out to learn more about PA, what it is, and what benefits it offers. In this post, I want to share and consolidate some of my findings -- hopefully this will be helpful to others who are considering or starting similar projects.

Now, before I get much further, credit where credit is due. The title of this post, "If You Predict It, You Own It", is a tag line I like, taken directly from Eric Siegel's Prediction Impact site. I recommend this site for an intro to the subject, as it offers many helpful articles as well as resources, such as the Predictive Analytics World conferences and training programs. 

So what is PA?

That will give you a good, quick intro to PA. What about the results? What's out there we can leverage to help sell such projects within our organizations? I did some of my own research into this and was also fortunate to have assistance from Sun's Digital Libraries & Research staff in locating a few additional publications. Here are some representative quotes/stats/images that make strong "sound bytes" in support of PA!

Optimizing Customer Retention Programs
by Suresh Vittal with Christine Spivey Overby and Emily Bowen, Forrester
October, 2008

  • "Marketers have long relied on analytical techniques to identify and reduce customer churn. For instance, segmentation models help marketers to better profile customers and understand behavior, while cross-sell and upsell modeling deepens relationships and creates barriers to exit."
  • "Marketers who target all types of respondents, not just the positives, risk wasting valuable resources on indifferent customers or at worst even triggering churn. This is especially critical in this climate of pressure upon marketing spend."
  • "Telenor found that by only targeting persuadables, it was able to reduce overall churn by 1.8%. A more telling statistic: These improvements were driven by only targeting 60% of the potential churners. The benefits of targeting smaller groups is clear — cost savings achieved from fewer contacts by telemarketing and lowering of customer fatigue through selective contacts." 
  • "The combination of increased retention rates and lower cost means Telenor will realize an 11-fold increase in uplift campaign ROI when compared with existing programs."
Turning Customer Interactions into Money
Peppers & Rogers Group
©2008 Carlson Marketing Worldwide.
  • "While the Internet and new technologies aren’t crystal balls, the sheer wealth of information that can be gleaned about today’s customers—and then applied toward anticipated future behaviors—is staggering. Failure to take this information into account is like leaving money on the table, or worse. You could simply hand it over to your competitors....Today’s smart companies use data, and the insight gained from it, to predict customer behavior."
  • "In one example, American Airlines used predictive analytics to better understand the relationship between various customer segments and differential flight patterns. They achieved sky-high ROI results of nearly 1,200 percent in a period of two months."
  • "IDC report studied dozens of companies and hundreds of predictive analytics projects. It found that the median ROI for the projects that incorporated predictive technologies was 145 percent, compared with a median ROI of 89 percent for those projects that employed only traditional analytics."
  • Nice summary chart from this article:
The ROI Cycle

"Mob Marketing" Webinar and Presentation
Suresh Vittal, Principal Analyst, Forrester Research
Jack Jia, CEO, Baynote
December, 2008

  • "Relevant and personalized interactions are critical for enhancing customer experience."
  • Baynote quoted the following benefits for their recommendation technology: 
    • 40% Lead Lift
    • 20% Net Revenue Lift (40% profit lift)
    • 400% Engagement Lift
    • 1000% Search Lift 

A vibrant and active amount of commercial activity also lends credence to the power and value of PA, and here's info on some PA providers:

And finally, just last week IBM bought perhaps the "granddaddy" of enterprise PA providers, SPSS, for $1.2 billion in cash -- a very serious endorsement of the power and value of PA! 

As noted, we are taking a DYI approach here, and you might be wondering about our results so far. I'll let you digest this info first, then follow up soon with a post on how SMILE is performing...

<script type="text/javascript"> tweetmeme_url = ''; tweetmeme_source = 'garyzel'; </script> <script type="text/javascript" src=""> </script>

Wednesday Apr 15, 2009

Update on Sun Web Personalization Initiative

I wrote an intro previously on SMILE, the Sun Machine Learning Engine, our new personalization system that utilizes predictive analytics. It went live in January, 2009, and we've been carefully measuring results since. So far, we are seeing higher click-through rates for SMILE served ads versus default ads, which is what we hoped would happen. But overall click through rates on these ads, whether personally targeted via SMILE or simply default, are very low, so we need more bang for the buck. We believe that the small ads served by SMILE in the right-hand navigation are easily overlooked -- many users simply ignore banner ads and/or the right hand column, concentrating on the central content of interest.

So our next step is to provide personalized recommendations in the body of the page. Here is a draft mock-up of a "carousel" type approach we hope to use:

Recommendations carousel

There are numerous challenges with this approach, as the page real-estate "above the fold" is highly valued, but that's where we need the recommendations to be if we really want them to be noticed and of value. A likely approach will be to make the carousel collapsible, so users can close it up if it's in the way, and then open it to look for personally recommended content. We'll also provide a new "All Recommendations" page that is solely dedicated to showing all recommendations for visitors.

We did a usability study recently to test these concepts and were pleased with the results. Customers told us:

  • They like what we're planning. We have literally millions of web pages, so if we can use our analytics to help customers quickly find the products, services, and content of most interest and value to them, we are saving them time and making them more efficient. Definitely a "win win" for all of us.
  • They would likely not notice the recommendations if they're not centrally located high on the page.
  • They don't have serious privacy concerns. Because we are an enterprise business site, they expect us to use analytics to improve the experience, and they'd appreciate the benefit. They would look at things differently if we were their financial institution, for example.
  • If the recommendations are not accurate or useful, they will quickly learn to ignore them. So the onus is squarely on us to ensure our system is up to the task of making accurate, valuable recommendations that meet our customers' needs.

This is an ambitious undertaking involving a whole bunch of teams across Sun, from Design to Publishing to Engineering to Analytics, so it's at least a few months away.  I'll keep you informed of progress and of course announce when you can check out your own personal recommendations!

Thursday Feb 05, 2009

New eMarketing Initiatives on

I've actually been on my new job for a number of months now, so it's time I start writing about it! After 10+ years working in the ESD (electronic software distribution) space here at Sun, it was time for a change, and so I took on a new challenge in the eMarketing realm. I am overall Program Manager on the web team for an initiative we call "Contacts to Revenue" (C2R). The essence of C2R is that we have zillions of web visitors, and especially downloaders, and we need to do a better job of identifying contacts and hopefully "nurturing" contacts into customers. There are several primary initiatives now underway.

First, we're creating new content we feel is of value to our customers and asking them to login to receive it. (If you don't already have a Sun Online Account, then there is a brief registration form that must be completed.) We feel this is a fair value exchange -- you tell us who you are, and we provide valuable information in return (examples include white papers, blue prints, webinars, and downloads). Here's an example:  Deploying Hybrid Storage Pools with Sun Flash Technology and the Solaris ZFS File System.

Second, I want to discuss Project SMILE -- The Sun Machine Learning Engine. We're building a new analytics engine to help provide more relevant content to web visitors. I'm the Project Manager for SMILE and pleased to say we went live with "phase 1" in mid-January. For the initial release, our goals were modest, focused primarily on improving the relevancy of small advertisements displayed on our main web site. We used a good ad banner server program previously, but it lacked sophisticated segmentation and targeting capabilities, so we built our own. If you are a return visitor to, we may serve you small ads that are more targeted to areas in which you've indicated an interest. All the ads are unobtrusive, appear in the right hand column, and are of this format:

Advertisement for xVM download 

For SMILE Phase 2, we're working on the ability to more proactively serve "Recommended Links" and other content of value based on web visitor's interests in our products and services.

We realize that asking for a login to obtain content and using machine learning and analytics to serve customers may be of concern to some. However, the bottom line is that we're in business to make a profit for our shareholders. Times are tough (duh), and it's imperative we do a better job of converting our web visitors to customers in order to grow our customer base and increase revenue. It's not a one way street -- we will continue to provide exceptionally valuable and unique content in return as well as striving to make relevant content more apparent and easier to locate on the web. 

In conjunction with these changes, we have recently updated our Privacy Policy to more accurately reflect how we'll be using customer data in support of these programs (among other updates). Please read it if you wish to be fully informed on this subject. (We also make it possible to opt-out of web personalization if you wish.) In any case, as this blog post and the new privacy policy illustrate, we're openly communicating about these new programs and changes. Our intent is to better serve our customers and our shareholders.


I helped design, build, and manage download systems at Sun for many years. Recently I've focused on web eMarketing systems. Occasionally, I write about other interests, such as holography and jazz guitar. Follow me on Twitter:


« July 2016

No bookmarks in folder