Tuesday Jan 26, 2010

Sun's Recommendations Carousel: Results and Learnings

In September, 2009, we released version 2.0 of the Sun Machine Learning Engine (SMILE), Sun's own advertising-oriented predictive analytics system. The primary customer facing manifestation of v2.0 is the yellow-bordered SMILE Recommendations "Carousel," which you see on many sun.com web pages:

SMILE Recommendations Carousel, small

As it's been more than four months since its release, I'll present some of the measured results. Overall, we've been very pleased with the value delivered by the carousel. (There have been some negative reactions, too, but I think that's to be expected when introducing highly-visible new advertising into the mix. More on that later...)

Bottom line, the carousel has exceeded our expectations and far outperformed the small ads we displayed using SMILE v1. Ads (also referred to as "recommendations") in the carousel were clicked by our visitors as often as one million times per week, yielding an average click through rate of over 3.6% (and the rate was much higher even on some sections of the site as opposed to others, depending on the audience). If you're familiar with online display advertising, you'll know that's a pretty phenomenal CTR, and we attribute it to the effectiveness of the SMILE analytics and recommendation engine, strong inventory of content and offers, and highly visible placement.

Clicking on an ad, though, is just the start, so we also measure what happens next, our so-called "key success metrics." These include:

  • Downloads Initiated
    Many ads point to either the Sun Download Center (encouraging visitors to get our free software) or specific software downloads, and SMILE drives on average 14-16,000 downloads per week. This is very beneficial, as getting our software in customers' hands is often their first real engagement with Sun.
  • Offers Obtained
    We provide many white papers that require simple registration or login to obtain, and the carousel often advertises and links to these offers (here's an example offer). They provide valuable contacts and leads to our Sales team, and SMILE drives on average 6-7,000 of these new contacts per week.
  • Teleweb widgetTeleWeb Success
    Here we measure how many visitors initiate a contact using the "TeleWeb widget" (pictured at right) after clicking a SMILE recommendation. Each such contact has a significant potential sales value (that we've measured accurately over time), so it's significant that SMILE delivers 100 or so contacts each week.
  • Sun Startup Essentials (SSE) Applications Submitted
    The SSE program is a win-win for Sun and startup companies, providing great value to each over the long run. SSE created a number of SMILE ads, and they perform very well, driving up to 25% of the weekly applications received into the program.

We measure many other variables and events, and in fact, thanks to recent enhancements, can now do closed-loop reporting with some of our Tele-Sales teams. This takes reporting one step further, providing the actual potential dollar value driven by SMILE. Here's how it works: When contacting a lead generated through a SMILE click, sales reps enter into their CRM system an estimate as to the potential value of the lead. This doesn't include all leads generated, as they go to different teams, and not all of them have enabled closed-loop reporting.  There is also a time lag between lead generation, dissemination, contact, and possible assignment of marketing pipeline value into the system. Even with those caveats, SMILE generated over $2,500,000 in potential leads value in December, 2009, alone! This is a great example of our push towards measurable, "deterministic" marketing and how we can realistically start to calculate ROI on this project.

Our real-world experience also comes with customer feedback, and we see room for improvement in the following areas:

  • First, do a better job of surfacing and explaining how the carousel and the recommendations work. The info has always been there but was "buried" on the All Recommendations Page. We would like to add an "About Recommendations" link to the bottom border of the carousel so that curious and/or concerned (with privacy) visitors can easily learn more about the program, how it works, and how we handle privacy related matters. The link would go to the bottom of the All Recommendations page where we've recently added a new section on recommendations, privacy, and program FAQs.
  • When a visitor closes the carousel, it stays closed for the duration of the browser session. But for users who visit often, this was inadequate. We would like to implement a different solution that keeps the carousel closed longer for users who don't wish to view it.
  • Due to an initial basic mapping system between images, products, and ads, we sometimes show duplicate images and/or lack image variety. We've been adding new images to the system to address this.
  • There are a few more simple enhancements that will also increase variety in the carousel, such as not showing slight ad variations for the same product or offer at the same time, and eliminating ads that link to the page the user is already on.
  • Lastly, on the back-end, we continue to refine and enhance the recommendation engine and methodologies with a goal of always increasing the relevancy of the ads to our visitors.

Will these enhancements see the light of day? Some decisions are pending the closure of the Oracle acquisition, so time will tell.

Regardless of what happens, though, I hope this information helps convey the strong results the program has produced, the success of our predictive technology, how it can be further improved, and the promise such systems hold for the future.

Friday Dec 04, 2009

Social Media Marketing Tactics for the Enterprise

Thanks to Frost & Sullivan for inviting me to their Web Experience Excellence 2009 Executive Congress held in downtown San Francisco on December 2, 2009. The specific topic was "Driving Strategic Online Advantage & Advocacy: Are your customers your evangelists?" and it was highly focused on how enterprises can and should use social media marketing. It pretty well covered the gamut from corporate blogs, wikis, and forums to Facebook and Twitter.

I'll give a quick run down on some of the most cogent "sound bites" and speakers of the day, but first a few observations. There was a lot of talk about "strategy" and is there such a thing as a "social media strategy." Some felt it was essential to develop one if not done already. Others felt that social media was just a part of the overall marketing/product strategy and not as such an end in itself. When considering the many examples presented of how social marketing is handled, it seemed the "strategy" answer ran all the way from none to formal. In some cases, it was (not surprisingly) more of a grass roots movement within the company, starting more with individual usage of social media, then moving up the "food chain" based on individual initiative and success. It's only recently that some companies have much more formally institutionalized the process, with dedicated social marketeers, teams, road maps, and indeed strategy. 

In all the case studies presented, there was almost no talk of "failures," as most of these initiatives, from grass roots to highly planned, have yielded positive results. In fact, this is such a hot and fruitful area that you may need to "just do it," even if the exec backing isn't all the way there yet. How and what to do were the focus of the presenters.

Alexander Michael, VP, Information & Communication Technology at Frost & Sullivan was our moderator and kicked off the first talk along with James Latham, Sr. VP, Strategic Marketing, at Open Text (which co-sponsored the event). Their talk on "Emerging Customer Expectations: Hyper-Technology and the Evolving Online Experience" set the stage for the day with some impressive stats on the growth of social media and its growing importance in the enterprise. 

Playing on the saying, "It's the economy stupid," they proposed the Web 2.0 version, "It's the people driven economy, stupid." Other key bullets from their talk:

  • Social media is not a fad. 
  • Don't communicate to GenY, communicate with.
  • Realize the new generation on the web has no loyalty. You must try to earn it with an engaging experience and seamless integration of social technologies. Three minutes is all you have...
  • You must encourage and not fear customer feedback. It's about Community, not Negativity. Most customer reviews are generally positive.
  • Don't try to fool people -- putting a web veneer over manual back-end processes won't work any longer.

Jeben Berg, YouTube Marketing Programs, Chief Innovationist, gave an entertaining and engaging talk, "Online on the Go: The Mobile Web and Impact of Video." He illuminated some clever YouTube marketing campaigns and offered great advice on how to succeed in that arena. I was impressed with his enthusiasm and domain expertise -- if you're developing a video campaign, you'll want to talk with Jeben. A few highlights:

  • Be who you are and be thick-skinned as well. YouTube won't take down parody or negative videos about your company or product -- they are a platform provider, not a content provider. The best response to a negative is your own video reply, done quickly.
  • Humor and irony rule. Emotion and competition resonate. "Low fidelity, high concept" works.
  • There's an audience for everything.
  • It's not about technology, it's about trust.
  • Videos should be self-contained -- you don't know where or when users are coming from.
  • Some examples:

Mark Yolton, Sr. VP, SAP Community Network, and Salim Ali, VP, Enterprise Solutions and Community Marketing at SAP, discussed, "Utilizing Social Media: Harnessing the Power of Online Communities for Loyalty and Advocacy." They talked about the evolution of communities at SAP and how they're used to Connect -> Collaborate -> Co-Innovate. They also talked about how to measure success, an important topic since typical "hard ROI" is often hard, if not impossible, to calculate with social programs. Their measurements center around number of members, traffic, contributors, and momentum. One interesting concept they raised was around segmentation, which is typically based on who you are. In the community space, what you're talking about is really important too -- segmentation by conversation.

Angela LoSasso, Global Social Media Strategy & Programs, HP, was up next for a discussion on "GenY and Beyond: Customer Experience in a Web 2.0 World." She talked about distinct benefits, such as how social can drive "Google juice" (think link relevance) and build brand relationships early on with younger (potential) customers. She was a proponent of crafting a social strategy, cautioning to first learn how your customers use social media before defining your approach. Another good point was not to confuse your strategy with the technologies and tools that enable it. Two other take-aways:

  1. Social and mobile are now one. (This was a recurring theme -- do not overlook your mobile customers!)
  2. Twitter can serve as an "idea factory."

James Latham spoke again next, talking about "Best Practices Live" and the Seven Essentials:

  1. Strategy with measurable goals
  2. Use all available inbound tools
  3. Content is king
  4. Be relevant
  5. Social media is here to stay
  6. Actionable analytics are the only metrics that matter
  7. Leverage your investments

James then introduced Christer Ljungdahl, Director, Web and Direct Marketing at National Instruments, who showed the "essentials" in practice at NI. Similar to SAP, there was a lot of focus on customer forums as a key interaction place for technically-oriented enterprises and their customers. There's certainly a cost avoidance benefit when customers provide each other tech support, and it's often faster and better than sitting on hold for tech support. But really these forums build community, and as Chris noted, "It's more believable when they say it" (think  product recommendations and reviews). His approach to building evangelists: Enable -> Share -> Listen -> Respond -> Recognize. One non-employee on their forum recently put up his 20,000th post! Forum managers observe a 90/9/1 rule: 1% of visitors are heavy contributors, 9% participate, and 90% visit.

It wouldn't be fair nor accurate to say they "saved the best for last," but Victor Cho, VP & GM, Consumer Internet & Software Services, Eastman Kodak, was an excellent presenter with clever, thought-provoking visuals. He engaged the audience in his talk on "Aligning Your Online Team with Corporate Objectives," and more specifically, "Eight ways to get your online team to truly deliver:"

  1. What do you measure?
    Typically the customer (like NetPromoter scores), shareholder (profit, market share, etc.) and employee satisfaction. To that, he added "Competitive Advantage." For example, note how Google focuses (some would say obsesses) on speed as an advantage.
  2. "Walls not screens" -- there's lots to measure, and focusing on one thing may miss the big picture. Perhaps your conversion rates are up, but sales/traffic overall are down. Profit might be up, but is it because you cut costs rather than increased the top line and margins?
  3. Measuring the customer -- figure out what causes (and how to measure) "wow" vs. "pain". Consider process vs. speed. Re-engineer process around newer/better customer experiences plus the need for speed.
  4. Conflicting speeds -- how often do you measure, yearly, monthly, weekly, daily, hourly, realtime? You need to measure different things at different intervals. Weekly feels right for many web metrics.
  5. Don't believe everything you see. Metrics don't always represent "the truth." There is still a very important place for judgment, testing, and experience.
  6. Gain from conflict, such as revenue vs. customer experience. He showed a formula, "$$$ = QV" where Q = quality and V = velocity (not value). Information spreads so quickly now that velocity is essential.
  7. Channel is another source of conflict -- you must balance and resolve channel conflict.
  8. Velocity of change -- when's the right time to push changes into the system? How do you balance quality versus speed? Will the change improve task completion? What changes will push your key metrics higher? You might think you're at the zenith, but then again, you might not be seeing the entire universe. No change, no gain.

So, with a final thanks to Frost & Sullivan for the nice drinks and hors d'oeuvre at the closing reception, that's a wrap.

Thursday Aug 06, 2009

If You Predict It, You Own It

As I've written about previously, I'm currently the Project Manager for the Sun Machine Learning Engine (SMILE) project, based on predictive analytics (PA) technology we're developing in-house. While I have a lot of experience building and managing complex web systems such as this, I haven't worked with PA technology before. I set out to learn more about it, for two reasons:

  1. Since I'm the PM for this project, it's generally a good idea to know what I'm doing and talking about!
  2. ROI is very important, both for this project and for ensuring the ongoing application of PA technology in general at Sun. This matters, of course, to Sun's management, and as you might imagine, it can't hurt to convey these benefits to our soon-to-be new owners as well. 

So, I set out to learn more about PA, what it is, and what benefits it offers. In this post, I want to share and consolidate some of my findings -- hopefully this will be helpful to others who are considering or starting similar projects.

Now, before I get much further, credit where credit is due. The title of this post, "If You Predict It, You Own It", is a tag line I like, taken directly from Eric Siegel's Prediction Impact site. I recommend this site for an intro to the subject, as it offers many helpful articles as well as resources, such as the Predictive Analytics World conferences and training programs. 

So what is PA?

That will give you a good, quick intro to PA. What about the results? What's out there we can leverage to help sell such projects within our organizations? I did some of my own research into this and was also fortunate to have assistance from Sun's Digital Libraries & Research staff in locating a few additional publications. Here are some representative quotes/stats/images that make strong "sound bytes" in support of PA!

Optimizing Customer Retention Programs
by Suresh Vittal with Christine Spivey Overby and Emily Bowen, Forrester
October, 2008

  • "Marketers have long relied on analytical techniques to identify and reduce customer churn. For instance, segmentation models help marketers to better profile customers and understand behavior, while cross-sell and upsell modeling deepens relationships and creates barriers to exit."
  • "Marketers who target all types of respondents, not just the positives, risk wasting valuable resources on indifferent customers or at worst even triggering churn. This is especially critical in this climate of pressure upon marketing spend."
  • "Telenor found that by only targeting persuadables, it was able to reduce overall churn by 1.8%. A more telling statistic: These improvements were driven by only targeting 60% of the potential churners. The benefits of targeting smaller groups is clear — cost savings achieved from fewer contacts by telemarketing and lowering of customer fatigue through selective contacts." 
  • "The combination of increased retention rates and lower cost means Telenor will realize an 11-fold increase in uplift campaign ROI when compared with existing programs."
Turning Customer Interactions into Money
Peppers & Rogers Group
©2008 Carlson Marketing Worldwide.
  • "While the Internet and new technologies aren’t crystal balls, the sheer wealth of information that can be gleaned about today’s customers—and then applied toward anticipated future behaviors—is staggering. Failure to take this information into account is like leaving money on the table, or worse. You could simply hand it over to your competitors....Today’s smart companies use data, and the insight gained from it, to predict customer behavior."
  • "In one example, American Airlines used predictive analytics to better understand the relationship between various customer segments and differential flight patterns. They achieved sky-high ROI results of nearly 1,200 percent in a period of two months."
  • "IDC report studied dozens of companies and hundreds of predictive analytics projects. It found that the median ROI for the projects that incorporated predictive technologies was 145 percent, compared with a median ROI of 89 percent for those projects that employed only traditional analytics."
  • Nice summary chart from this article:
The ROI Cycle

"Mob Marketing" Webinar and Presentation
Suresh Vittal, Principal Analyst, Forrester Research
Jack Jia, CEO, Baynote
December, 2008

  • "Relevant and personalized interactions are critical for enhancing customer experience."
  • Baynote quoted the following benefits for their recommendation technology: 
    • 40% Lead Lift
    • 20% Net Revenue Lift (40% profit lift)
    • 400% Engagement Lift
    • 1000% Search Lift 

A vibrant and active amount of commercial activity also lends credence to the power and value of PA, and here's info on some PA providers:

And finally, just last week IBM bought perhaps the "granddaddy" of enterprise PA providers, SPSS, for $1.2 billion in cash -- a very serious endorsement of the power and value of PA! 

As noted, we are taking a DYI approach here, and you might be wondering about our results so far. I'll let you digest this info first, then follow up soon with a post on how SMILE is performing...

<script type="text/javascript"> tweetmeme_url = 'http://blogs.sun.com/gaz/entry/if_you_predict_it_you'; tweetmeme_source = 'garyzel'; </script> <script type="text/javascript" src="http://tweetmeme.com/i/scripts/button.js"> </script>

Wednesday Apr 15, 2009

Update on Sun Web Personalization Initiative

I wrote an intro previously on SMILE, the Sun Machine Learning Engine, our new personalization system that utilizes predictive analytics. It went live in January, 2009, and we've been carefully measuring results since. So far, we are seeing higher click-through rates for SMILE served ads versus default ads, which is what we hoped would happen. But overall click through rates on these ads, whether personally targeted via SMILE or simply default, are very low, so we need more bang for the buck. We believe that the small ads served by SMILE in the right-hand navigation are easily overlooked -- many users simply ignore banner ads and/or the right hand column, concentrating on the central content of interest.

So our next step is to provide personalized recommendations in the body of the page. Here is a draft mock-up of a "carousel" type approach we hope to use:

Recommendations carousel

There are numerous challenges with this approach, as the page real-estate "above the fold" is highly valued, but that's where we need the recommendations to be if we really want them to be noticed and of value. A likely approach will be to make the carousel collapsible, so users can close it up if it's in the way, and then open it to look for personally recommended content. We'll also provide a new "All Recommendations" page that is solely dedicated to showing all recommendations for visitors.

We did a usability study recently to test these concepts and were pleased with the results. Customers told us:

  • They like what we're planning. We have literally millions of web pages, so if we can use our analytics to help customers quickly find the products, services, and content of most interest and value to them, we are saving them time and making them more efficient. Definitely a "win win" for all of us.
  • They would likely not notice the recommendations if they're not centrally located high on the page.
  • They don't have serious privacy concerns. Because we are an enterprise business site, they expect us to use analytics to improve the experience, and they'd appreciate the benefit. They would look at things differently if we were their financial institution, for example.
  • If the recommendations are not accurate or useful, they will quickly learn to ignore them. So the onus is squarely on us to ensure our system is up to the task of making accurate, valuable recommendations that meet our customers' needs.

This is an ambitious undertaking involving a whole bunch of teams across Sun, from Design to Publishing to Engineering to Analytics, so it's at least a few months away.  I'll keep you informed of progress and of course announce when you can check out your own personal recommendations!

Thursday Feb 05, 2009

New eMarketing Initiatives on www.sun.com

I've actually been on my new job for a number of months now, so it's time I start writing about it! After 10+ years working in the ESD (electronic software distribution) space here at Sun, it was time for a change, and so I took on a new challenge in the eMarketing realm. I am overall Program Manager on the sun.com web team for an initiative we call "Contacts to Revenue" (C2R). The essence of C2R is that we have zillions of web visitors, and especially downloaders, and we need to do a better job of identifying contacts and hopefully "nurturing" contacts into customers. There are several primary initiatives now underway.

First, we're creating new content we feel is of value to our customers and asking them to login to receive it. (If you don't already have a Sun Online Account, then there is a brief registration form that must be completed.) We feel this is a fair value exchange -- you tell us who you are, and we provide valuable information in return (examples include white papers, blue prints, webinars, and downloads). Here's an example:  Deploying Hybrid Storage Pools with Sun Flash Technology and the Solaris ZFS File System.

Second, I want to discuss Project SMILE -- The Sun Machine Learning Engine. We're building a new analytics engine to help provide more relevant content to web visitors. I'm the Project Manager for SMILE and pleased to say we went live with "phase 1" in mid-January. For the initial release, our goals were modest, focused primarily on improving the relevancy of small advertisements displayed on our main web site. We used a good ad banner server program previously, but it lacked sophisticated segmentation and targeting capabilities, so we built our own. If you are a return visitor to sun.com, we may serve you small ads that are more targeted to areas in which you've indicated an interest. All the ads are unobtrusive, appear in the right hand column, and are of this format:

Advertisement for xVM download 

For SMILE Phase 2, we're working on the ability to more proactively serve "Recommended Links" and other content of value based on web visitor's interests in our products and services.

We realize that asking for a login to obtain content and using machine learning and analytics to serve customers may be of concern to some. However, the bottom line is that we're in business to make a profit for our shareholders. Times are tough (duh), and it's imperative we do a better job of converting our web visitors to customers in order to grow our customer base and increase revenue. It's not a one way street -- we will continue to provide exceptionally valuable and unique content in return as well as striving to make relevant content more apparent and easier to locate on the web. 

In conjunction with these changes, we have recently updated our Privacy Policy to more accurately reflect how we'll be using customer data in support of these programs (among other updates). Please read it if you wish to be fully informed on this subject. (We also make it possible to opt-out of web personalization if you wish.) In any case, as this blog post and the new privacy policy illustrate, we're openly communicating about these new programs and changes. Our intent is to better serve our customers and our shareholders.


I helped design, build, and manage download systems at Sun for many years. Recently I've focused on web eMarketing systems. Occasionally, I write about other interests, such as holography and jazz guitar. Follow me on Twitter: http://twitter.com/garyzel


« July 2016

No bookmarks in folder