Wednesday Nov 12, 2008

Web 2.0 Summit 2008, Day Two

(This is a continuation of my prior post, "Overview of  the Web 2.0 Summit 2008.")

Day Two started with Paul Otellini, President and CEO of Intel. He focused on two big areas of opportunity that Intel sees. The first was "enterprise" (or professional) social networking. He showed a thought-provoking demo of the unified enterprise community network of the future -- tightly integrated communication tools, professional learning management, robust search and RSS feeds. The demo focused on a fictional Intel new hire in Asia and how such an enterprise web community would smooth her integration into the company and also accelerate her learning and effectiveness. Then he said something like, "Of course, these tools don't exist yet..." 

The second big opportunity was the "personal Internet," where the machine brings relevant and timely info to you rather than you having to seek it. This led to another demo, this time of the smart mobile device of the future. Features included machine translation (written and verbal) in the palm of your hand, instant commerce, lots of geo-awareness, etc. It demonstrates Intel's focus on bringing the power of the web to mobile devices.

Next up was Kevin Kelly, a long time reporter on the culture of technology who helped launch Wired Magazine. This is more or less what his first slide looked like:

The number 6527

Care to guess the meaning?

Turns out it had been exactly 6,527 days since Tim Berners-Lee transmitted the first web page. The Internet (r)evolution since then has been amazing, but Kelly really said we need to hasten the next phase, the so-called Semantic Web. It's about awareness and linking of data. First we linked computers, then pages, and next we need to link data. 

So what will the next 6,500 days bring? Kelly feels it'll go way beyond "the web, only better." He spoke of one large machine of interconnectedness, the Web Operating System (I couldn't help thinking of Sun's seminal mantra, The Network is the Computer). The web will "own" all data -- if it's not part of it, it doesn't exist. All devices are connected, processed, and structured. One media platform for all. Everything is always on. "Believe in the impossible."

This could be an awesome thing, though it does have that "big brother" ring to it as well. Let's hope the "security guys" get it right.

The plethora of iPhone enthusiasts in the audience were entertained by a talk with Ralph De La Vega, CEO of AT&T Mobility and Consumer Markets. Let's just say he's pretty happy with how things are going. 

The web can and will play a huge role in the future of health care, as discussed in the "Future of Health" panel. Lots of talk of genetic testing and online communities. Check out 23andMe, then order your own genetic profile for a mere 400 bucks.

Appearing in full battle fatigue, Army CIO Jeffrey Sorenson showed the military side of social media. The Army lags behind the speed of commercial innovation and is just rolling out Web 2.0 now. Security concerns are paramount of course. But we saw interesting demos of "Battle Command Knowledge System" and "Command Post of the Future" -- web based systems that will greatly enhance our military capabilities and speed/accuracy of communication.

The "Future of Media" panel provided an engaging discussion featuring Evan Williams (Twitter), Joel Hyatt (Current TV), and Ken Auletta (The New Yorker). I remember coming out of my first Web 2.0 Expo in 2005 thinking I had to start a blog (which I did). This time, it was all about Twitter, so I'm going to give it another try too! (I'm admittedly not 100% convinced, but time will tell.) They talked about Twitter changing how people connect --> will it actually change our culture over time? 

Today we are enabling "conversations" in every possible format, though the questions remain on how to monetize these new interactions. (They didn't have the answer, sorry.) 

I found Hyatt's thoughts on Current TV provided a refreshing perspective:

  • Many people innovate, then look for a business model. They did the opposite.
  • They're developing "user generated advertising" (he showed a clip of a Toyota commercial made with a hand held video camera). Their study shows these ads are "preferred 9 to 1."
  • The networks are focused on moving TV to the Internet. Current is focused on moving the Internet experience to TV. 

Next speaker, Saul Griffith, a rocket scientist type for sure, reminded us that we all contribute to the energy crisis. We all have to help solve it. He's built a very clever web site that lets you easily determine how much energy you use in a day. He suspects we'll all be surprised and ultimately better conservationists after finding that out.

I have a 17 years old son, so couldn't help noticing the physical similarities to Mark Zuckerberg, founder of Facebook. Yes, he's young, but he was very composed and well-spoken. (BTW, it's been a big week for 24 year olds around here!) He took some heat for the closed nature of the Facebook platform, but that's changing, and he said there is a natural progression from closed to open systems over time. (That's a key part of Sun's open source vision, so let's hope he's right.) He spoke of a long term goal to enable people to share information any way they want on any device. How to make money? Think of how to best entice customers to share information, then deliver "engagement ads" that are highly effective. 

A very high powered (pun intended) panel was up next on Cloud Computing -- Paul Maritz, CEO of VMware, Marc Benioff, CEO of salesforce.com, Kevin Lynch, CTO of Adobe, and David Gerard, Google. You know this is one of the hottest topics of '08 and extremely well covered, so I don't think there was a lot of news coming out of this. Some interesting concepts were mentioned though:

  • Information will outlive devices and applications; the cloud becomes the "information bank."
  • Providers have to protect, enable access, and add value to information stored with them.
  • We're hitting the "fourth generation" of software, getting back the richness of client-server applications (which we lost with Web v1.0) but keeping the power of the Internet.
  • They need to provide the tools to enable the movement of apps to the cloud.
  • We need to break down the disconnect between rich customer experiences and pedestrian enterprise experiences.
    • What if you had a rich Flash interface to SAP? (Adobe wants to be the "interface to the cloud.")
  • It doesn't have to be a low margin business. Differentiate by providing unique value and/or a large ecosystem.
  • Clouds are still proprietary, though. More value will be created as they interoperate and provide new services. (We're seeing this now with Salesforce's integration with Google Apps, for ex.) 
    • Standardization is needed, as well as new data models that break down data silos.
  • Eventually, consider we'll have "personal clouds" as opposed to multiple friend lists on separate web sites.

Shifting back to the future of advertising, there was a talk with Jack Klues of Vivaki -- they control billions and billions of dollars in ad budgets for some of the world's biggest brands. His message for weathering the economic storm:

  • Maintaining and protecting brand is at least as important during a downturn as any other time.
  • You can increase share as media buys become cheaper.
  • Don't lose sight of what people care about; keep on message and brand.
  • Continue to look at how to optimally blur the lines between advertising and content (ethically, of course...)

A "Launch Pad" event was next. Five start-ups gave their elevator pitch to a panel of VCs. The audience voted on our favorite by texting votes which were tabulated and projected real time. I agreed with the winning choice, GoodGuide. Great concept and valuable info -- take a look. (No real losers here, all were of interest but not sure they'll all survive: Predictify, Sungevity, Everyscape, Qik.)

The day closed with a Music discussion with Chris DeWolfe, co-founder of MySpace, and Edgar Bronfman, CEO of Warner Music. It was a bit too much focused on the launch of MySpace Music I thought. They said the music industry will be "very different in 5 years," but not necessarily how. And with that, another long but stimulating day ended.

Continue to read about Day 3...

Tuesday Nov 11, 2008

Overview of the Web 2.0 Summit 2008

I felt fortunate to attend my third Web 2.0 Summit, November 5-7, 2008, in San Francisco (thank you, Andreas!). I really thought it was an excellent event and fairly different from the prior ones I attended (here are my comments on the 2005 Conference and 2006 as well.) What was most different was the strong and well-coordinated thematic focus on "World Meets Web" -- it truly permeated and unified the sessions and discussions. The concept behind this is that we've come a long way with the web and it has become a major influencer on the global society. How do we use this new found medium for the betterment of the planet and to deal with all the issues we face? It couldn't have been a more timely theme,  nor punctuated better than Al Gore's closing comments, as he noted Barack Obama could not have been elected without the web! Talk about web meets world....

The format was generally the same as previously. Wednesday morning provided the only workshops, where you have the choice of attending several tracks that run simultaneously. (This is where you typically wish you could be in two or more places at once so you don't miss anything.) Starting Wednesday afternoon and through the conference closing late Friday, we all assembled in a large conference room for the engaging interviews with true industry leaders as well as the “high order bits” (short, 10-20 minute talks by various domain experts).

There's lots of info online of course about the Summit, and in the interest of time and efficiency, I'm not going to bother deep linking to everything. Here are the bios for all the speakers, and many of the presentations (and video) are online too. My main goal is to share some of the insights I captured from the various speakers and events.

For the workshops, I chose topics related to presentation and advertising, as this relates closest to what I'm working on now (developing new web personalization and ad serving capabilities for Sun).

The first talk was a bit of a strange start, as a little ways into it I realized it had nothing to do with the web (at least directly). Go figure. Nancy Duarte's topic was "Telling Meaningful Visual Stories" and was really about creating compelling presentations. It was targeted partly at least to the many entrepreneurs that attend, with lots of guidance on how to make compelling pitches to VCs. But I think there are lessons to be learned for presenting info on the web as well. Some key takeaways:

  • To connect with the customer, become a story teller. Share experiences, be vulnerable perhaps, believe that your own story can touch customers. (This is something that cannot be outsourced!)
  • Consider your audience first and how to reach them, then develop the content. A presentation is not a “document”.
  • Break out of the mold – start with a blank page, not a template. Change your environment, get a different perspective, collaborate and brainstorm on ideas. You want to inform, inspire, persuade.
  • Avoid complicated architectural diagrams, as they don't resonate. "Tell" your architecture as a story.
  • Use metaphors. Include conflict to grab attention. Produce ideas and stories, not slides. Be transparent, be different. Your stories can change and influence!

The next workshop I attended was "Consumer Brands Tackle Marketing 2.0," with some high profile marketers from leading companies. Some nuggets from that panel:

  • Technology changes, people don't.
  • Social media offers new opportunities for gaining customer insight, if you know what to do with it. Consider "perpetual marketing," as the engagement doesn't end. We're still evolving how to use social media consistently in marketing programs.
  • The audience defines the message. Good creative still wins, technology can't overcome crappy messaging and content.
  • Traditional media has the reach but many in the audience won't be interested. For example, if you advertise Pampers on national TV, what percentage of the audience cares? This is really where we can differentiate advertising on the web, by targeting interested and relevant audiences much more closely.

As I mentioned, we're working on more personalization initiatives here at Sun, and one outcome is that we're bumping up against our own privacy policy. There's a fine line there about what we can and cannot do. I asked the panel about how they deal with it, and mostly got shrugs! David Knox from Procter & Gamble did say they had just changed their privacy policy to enable global opt-in/opt-out on their site, instead of by section as it was previously. Their approach was to email every single customer and inform them of the new policy.

So it seems everyone is trying to deal with this, and there is no perfect answer. I think the best thing is to be transparent with what we're doing. We need to revisit Sun's privacy policy to see how it can evolve to match the evolution of our web and personalization capabilities. When we do make a change, we'll need to communicate it appropriately.

The last workshop I attended had the not-so-helpful title of "Please - Not Another Network Panel." Sponsored by Yahoo, it actually focused on web advertising and ad networks. I'm pretty new to this area so didn't follow all the subtleties of the conversation about how these mega-networks work, but I managed to learn more about that side of the industry.

And with that, it was lunch time, then we headed in for the start of the mega-conference. The focus on web meets world was clear from the start, as the first interview was with Larry Brilliant, head of google.org, Google's philanthropic arm. Google committed from the start to putting 1% of revenue, profit, and resources into philanthropy. We'll see this theme repeated throughout the conference – the web has made you millions (or billions), so how will you use that to make the world a better place?

Google is focused on three key initiatives:

  • Inform & Empower -- mostly a third world focus on empowering communities to get better information and accountability from their governments.
  • Predict & Prevent – Early disease identification and prevention. This touched on another recurring theme, “instrumenting the world.” We have great new sensor capabilities and can tie the sensors together via the web for almost real-time, unprecedented insights into the global ecosystem.
  • Renewable Energy -- specifically, we need a form of renewable energy that is cheaper than coal. At that point, there is no financial barrier to adoption.

Bottom line: "It's time to use technology to make the world a better place."

Mary Meeker gave her usual dizzying blitz of a zillion financial slides in about 15 minutes. It's pretty hard to digest that quickly but my takeaway was that, while the economy may be tanking, there is still plenty of growth and opportunity for Web innovators.

Rajesh Jain from Netcore Solutions (see his blog at emergic.org) explained how there are about 10 times as many people with cell phones in India as have Internet access. As a result, SMS is huge, offering ease of use, low cost, and easy value added services. There's a huge direct to consumer reach. He used the term “invertising” for invited ads in SMS messaging.

Up next was super-VC John Doerr, who helped fund Sun (amongst many other companies) years ago. He talked about Obama being the first President who will have a CTO, and suggested (Sun co-founder) Bill Joy would be a perfect fit. He felt energy research is our #1 priority and expressed a big concern that we're not graduating enough scientists and engineers. He also felt it was absurd that we train foreign students, then kick them out of the country once they're trained. I loved the way he summed it up:  "We should staple a green card to their diplomas!" Makes sense to me.

Doerr offered up survival tips for start-ups during the economic meltdown (which he described as more than bad mortgages, it's a "crisis of confidence"). I think much of this is great advice regardless of how big your company is:

  • Hunker down, take on the long view, focus on your core
  • Act now, and with haste (before funding gets cut)
  • Protect the core of your business (don't lay off the key people!)
  • Have 18 months worth of cash in the bank
  • Defer facilities investments
  • Re-evaluate R&D priorities
  • Renegotiate all your contracts
  • Everyone in the company needs to SELL
  • Give equity instead of cash
  • Secure the cash (i.e treasuries, somewhere safe)
  • React quickly to leading indicators and KPIs
  • Over communicate, don't sugar coat bad news
He was quite bullish on the iPhone and said they're seriously investing in it as a platform. He likes cooliris.com. He likes companies focused on the conversion of IT and "green technology."

Next up, Tony Hsieh, the personable CEO of Zappos.com. I think the Zappos story is well-publicized so won't go into details, other than to say it's quite impressive what they've done. This web-based "shoe store" is now a $1B business and expanding quickly beyond footwear. A few key points:

  • The company is "powered by service".
  • The goal is to WOW each & every customer.
  • Business is built by repeat customers and word of mouth.
  • It's good to have people call your customer support line. There is no better branding opportunity then speaking with your customers.
  • Company culture = company brand.

Larry Lessig, the outspoken Stanford Law professor, talked about the current failures of our government. He says it's come to the point where the Number One job of our elected representatives is to get "tenure" (i.e. raise enough money to get elected over and over again). Money destroys trust. We must change congress to fix our economy.

Day one ended with Yahoo CEO Jerry Yang on the hot seat, grilled somewhat mercilessly (but fairly, I think) by conference promoter John Battelle. Jerry took the hard questions graciously and stood his ground. He explained Yahoo's strength as a consumer brand that enables people to get what they want from the 'net. It's the starting point.

He talked about recent efforts to open up the Yahoo platform with the Yahoo! Open Strategy, enabling third party apps to integrate with Yahoo services. There is opportunity for all in the creativity that will come out of opening up the platform.

What I hadn't realized was the large press presence and how they would pick up on what was being said. While discussing Microsoft, Yang said he thought Microsoft should buy Yahoo. I was surprised when this casual statement during a conversation turned into national news.

As you can see, it was quite a stimulating Day 1!

As an aside...
I was amazed at the proliferation of iPhones in the audience (and Blackberries -- the lady sitting next to me had one of each on her lap) and somewhat dismayed at how many people were looking down constantly with an eerie blueish light on their faces from the phone displays. It didn't seem to me they were paying attention to the conference, and I wonder about this newest contributor to attention deficit disorder. But perhaps they're just better at multitasking then I am.

Continue to read about Day 2...

Friday Oct 24, 2008

Download Best Practices, Part 3

Here's a "best practice" that is a source of never-ending debate: Should we require login/registration in order to download our products? There is no right answer, but there are plenty of pros and cons.

Sometimes, we have no choice but to require login in order to comply with government export regulations. (For example, if a product contains some types of strong encryption, we must keep a record of who has obtained it.) An obvious reason in favor is that knowing who downloads our software is of great value in follow up marketing initiatives as well as demographic analysis of our customer base. It's not all a one-sided equation though. For our customers who login first, we offer a useful download history function, and we also automatically provision our My Sun Connection portal with a list of products of interest to you. (This automatic portal provisioning is set up for many, though not all, of our downloads.) It's very handy, as you can click on product names and quickly see a valuable list of related resources and information. (I encourage you to check out the portal if you're not familiar with it. Once there, be sure to click the "My Products" tab to see how we aggregate product information from many resources in one place!)

The other side of the coin is that requiring login or registration before downloading definitely hurts the conversion rate. Using our web analytics, you can see users drop off with every extra click, and the largest abandonment is at the login/registration step. In many cases, the value of getting our software into the hands of our (often first-time) customers as quickly and with as few roadblocks as possible trumps the value of collecting user information, so we do not require registration. 

With such conflicting agendas, you must be wondering by now what I'm going to recommend as the best practice? Well, that's easy, don't use either -- use optional registration! This feature provides a middle ground that works for most products -- just give customers the choice. If they want to login to gain the benefits offered, we welcome and appreciate it and believe it's worthwhile. And if the customer just wants the software as quickly as possible, there's no need to register, just click Continue. Here's what you'll see when you request an "opt reg" product on Sun Download Center:

optional registration interface

For those who do choose to login first, we find a better quality of data since they are doing so of their own volition, and it allows us to build a stronger relationship with the customer. Of course, we have to face the facts -- a majority of customers will skip the login step and click Continue. For that reason, we highly recommend building optional product registration into the product's installer as well. Once you have committed to installing the product, there's an even stronger opportunity to build the relationship and help support the software on your system.

Fortunately we built our download system to support all three options (required, optional, none) easily, so we can meet whatever the product team wants with the flip of a bit in the product database. However, our recommendation remains to use optional registration whenever possible.

This is the third in a series of download best practice posts.

Friday Oct 03, 2008

Two new bloggers now writing about downloads


After more than 10 years of focus on download systems here at Sun, I was ready for a change of pace (as you might imagine). Once we got our new download system out the door, my manager was much more willing to consider a change for me, and then a great opportunity arose. I'm now managing a new project that is developing a machine learning engine for better targeting our offers to our customers. While the obvious intent is to benefit Sun by increasing the uptake on our products and services, we feel it benefits customers as well by providing more relevance to the ads and content we provide you on our web sites.

It's pretty much a whole new world for me, but I'm enjoying the challenge and learning about a whole new area of analytics. It would be a shame to ignore the experience I've acquired in the ESD realm, so I'm still participating in download strategy discussions and helping out where I can. I may well blog further on the subject as time allows.

But in any case, I am happy to see two new blogs about downloads and ESD that have sprung up recently. As I wrote in my very first blog posting three years ago, "I haven't found many resources for us 'ESD Professionals.' I'm not aware of trade journals nor web sites dedicated to ESD. I always enjoy meeting and speaking with my peers at Sun and other companies but wish there was more interaction, more sharing of best practices, and the myriad other benefits that come from knowledge sharing in a particular field. This blog is intended to be a small outreach..."

Well, I can't say I've seen an explosion in content, but I heartily welcome two new contributors to the ESD blogosphere!

  • The overall Project Manager of the new download system was Alfred Chen, and he's the guy in charge now that I've stepped into a new role. Alfred is leading our Download Business and Strategy Teams, and we are very lucky to have him in those roles. Working with Alfred very closely for many years, I can only say he's a tireless contributor with exceptional knowledge, skill, and dedication. (And he used to be the manager of our download engineering team, so he's really one of those great persons with deep IT and Business experience.) So, to get to the point, Alfred's started blogging lately (about a variety of subjects) -- check it out.
  • OMS Safe Harbor is one of the few companies out there with a major focus on electronic software delivery, and they've built up an impressive set of capabilities, products, and services. They recently let me know about their new ESD blog, and I've already read a number of interesting articles there. Great to see this outreach from OMS.
I hope you'll find these new bloggers of interest!


Tuesday Jul 08, 2008

A Standard for Measuring and Reporting Downloads


Mozilla and Firefox have garnered a lot of well-deserved publicity for their new Guinness record for most downloads in a day. This raises the question (again) of what constitutes a "download" and how is it measured? (I first touched on this subject a couple of years ago, writing "When is a download a download?".) I was glad to see in the above referenced BBC article, "The official figure was confirmed after logs from download servers were audited and checked to ensure duplicate and unfinished downloads were not counted." So that implies "attempted" downloads weren't counted, only completed. That's good, as I'm sure some stats we see reported don't make such distinctions. 

So with Guiness in the mix now, and with all the focus on downloads in the free & open source software world, I propose it's time for an agreed upon standard on how the number of downloads is reported. (To be honest, I'm uncertain if this could ever be a "formal" standard and who would govern it -- please let me know if you have any suggestions.) For starters, here's what I propose:

  • Simple case: One Product = one file
    • This case means that the entire product is contained in a single downloadable file. If it is downloaded successfully, the customer can install and use the product.
    • The suggested measurement is the one generally used -- the last byte of the file is delivered to the customer.
      • Note this may not be 100% accurate. Threaded download managers can conceivably download the last byte of a file and then fail an earlier byte range before the entire download completes. But this is an edge case. It would also be extremely resource intensive to have to match up all the byte range requests in a threaded download scenario to "prove" that one user completed the download. Thus, I think the "de facto" standard of last byte delivered is acceptable.

  • Complex case: One Product = multiple files
    • First, make the distinction between required and optional files:
      • Required files are required to install the product. There may be one or more required files as part of the download. 
        • For example, on Sun Download Center, there is an option to download Solaris OS as 5 CDs. If you do not download all five, you cannot install the product. Thus, all five are required files, and each one must be downloaded to install Solaris.
      • Optional files are not required to install or run the product. Optional files may consist of "plug-ins,"  "add-ons," or "modules" that add functionality to the base program. Also included would be documentation, checksum files, additional language packages, and the like.
    • The measurement of a completed download in this case is that all required files are downloaded by the same customer. (This can be a challenge to measure, but it can be done in most cases with a good degree of accuracy.)
    • Optional files do not count and should never be reported as "product" downloads.

  • In either case, "Attempted Downloads" should never be reported as a product download. 
    • For a single file product, an attempted download is when the user starts the download but does not receive the last byte of the file.
    • For complex products, an attempt is when any individual required file is not completed and/or the customer does not successfully download all required files.

Monday Jun 02, 2008

Download Best Practices, Part 2


I'm going to list three more of the download best practices that we encourage all our product teams to use as they develop products for electronic software distribution (ESD). This is the second part of a continuing series I'll publish from time to time. (You can see Part 1 here.) Some of these suggestions may seem like "no-brainers" (at least once you understand the basics of ESD), but they need to be stated and documented nonetheless and are certainly helpful to teams that are just starting out to prepare downloadable software.

1) Compress the files as much as possible. Obviously smaller files are cheaper/faster for everyone to download, and faster delivery means less time for things to fail and higher completion rates. Generally, we recommend using zip compression, as it's pretty much a de facto standard at this point (and is supported on Solaris OS as well). There are many free and open source distributions product teams can use, such as Info-ZIP, bzip2, and 7-Zip. You should experiment to see which one provides the best compression (in one test here, 7-Zip outperformed the others by about 20%, but your results may vary). There are also proprietary compression programs ("NOSSO," for example) that can greatly outperform "vanilla" zip compression, but they come with a price tag. If your download size really really matters, they may be worth investigating.

2) Use standard MIME types for file name extensions, for example "filename.zip" and "filename.exe." Web servers, browsers, download managers, and many other pieces of the Internet infrastructure rely on standard MIME types to ensure download transactions work correctly and consistently. So, don't release files without a MIME type extension (for ex., use "README.txt," not "README"), and don't make up file extensions or use non-standard ones, such as "filename.aa," "filename.bb," etc.

3) For large files, offer alternatives. Not everyone can successfully download a large file, especially ones larger than 2 GB. Other choices should be provided, such as a segmented ("chunked") version of the large file, CD images instead of a single DVD, and/or the option to order free or low cost CDs or DVDs. See my earlier post on "Breaking the Large File Barrier" for more details.


Thursday May 15, 2008

Using wget With Our New Download System


Soon after releasing our new download system, we received comments that the command line download client wget (and others) no longer worked. This is a result of a new session-based security model used to protect download links from being copied and re-used (which could circumvent numerous Legal requirements necessary to authorize a download). 

We have implemented a work-around now, so I just wanted to put this quick note out there to increase awareness of its availability. If you'd like to use wget or similar tools with Sun Download Center, please review the work-around, posted on our new Downloads FAQ Wiki.

Sorry for any inconvenience this has caused -- it was one of those unintended consequences. We believe we have a solution for it and will implement it as soon as we can. In the meantime, I'm pleased we could at least offer a temporary fix.

Get the work-around script here: Using wget with the Sun Download Center


Friday Apr 18, 2008

Video Demo of Sun's New Download System


OK, so things took a bit longer than we expected, and we hit a few bumps in the road, but our new download system is now fully operational and humming along! Just this week we finished moving all sun.com downloads onto it, and we're extremely pleased to reach this milestone.

To help introduce the new system (and also get newcomers and experienced users alike downloading as quickly, easily, and successfully as possible), please check out our new demo video. It features a walk through of the Solaris OS download experience, as well as helpful tips on using Sun Download Manager. The video takes less than five minutes.




I hope you enjoy the video, and more importantly, that you benefit from the enhancements we've made to downloading from Sun. As always, your suggestions for further improvement are welcome. You can leave me a comment, or if you encounter any questions or issues, please let our customer support team know. Thanks!


Friday Mar 21, 2008

Download Best Practices, Part 1


We've been working on a new internal wiki to capture download best practices gleaned over the past 11 years of designing and managing our high capacity download systems. I think this is good and valuable information so wanted to blog about some of our findings. This will be the first in a series of posts on the subject.

First, let's consider our goals in defining and publishing these practices. At a high level, I'd sum them up as follows:

  • Ensure customers are able to download successfully, as fast as possible and with minimal hassle.
  • Provide a consistent and efficient download experience.
  • Increase completion rates and reduce abandonment.
  • Ensure our systems and processes optimize the conversion of downloaders into (paying) customers.
  • Save time and bandwidth (cost) for Sun and our customers.
  • Within Sun, document the processes to release software for download, and ensure they are followed.

Our first best practice is seemingly simple yet frankly it's not been followed consistently over the years: use a unique file name for every file in every product and version release. There are several reasons why this is so important, the first of which is unique to our systems:

Sun Download Manager uses special "verification property files" (VPFs) to checksum files as they are downloaded from Sun Download Center. The VPFs are mapped to unique file names. So, here's where we get into really deep water if file names aren't unique from release to release! If we release a product with file name "product.zip", a VPF will be created called "product.zip.sdm.zip" that SDM grabs automatically and uses to checksum the file and ensure a successful download. Now if the product team releases the next version of the product but does not update the file name, a new VPF will be created with the same name as the prior one. This works fine for downloading the new product. But when users try to download the prior version, there will be a checksum mismatch between the old file and the new VPF -- the download will fail. And that's why this is so critical on our system.

Here are a couple of other good reasons (not unique to Sun's download systems):

  • Downloads often flow through content distribution networks which cache files around the world for fastest delivery. Unique file names ensure caching works most efficiently and eliminates the potential for delivering incorrect content from the caches.
  • Unique file names reduce potential errors when locating, copying, moving, updating, reporting on, and archiving content.

It's really not hard to do, so we hope you'll follow this practice!

If you work for Sun, you can view the full Download Best Practices Wiki.

Sorry, this is for Sun employees only. The link will not work unless you are inside Sun's firewall.


Thursday Feb 14, 2008

New Wiki for Sun Download FAQs


I've been blogging for a while now, and I guess it was inevitable that wikis were in my near future as well! Having spent a lot of time and energy learning HTML and web publishing, I was hesitant to dive in and learn "wiki language" and the new publishing method. But a little encouragement from our VP goes a long way -- Curtis suggested we take a stab at a wiki to involve more of our community in the areas of download support and information.

So, we've taken the plunge and have posted a new Sun Downloads FAQ on wikis.sun.com. We already have a comprehensive FAQ devoted to Sun Download Center, and there's no point in repeating all the content in a wiki format. Rather, we want to broaden the scope to include all Sun downloads (open source products, for example) and related areas. I think we should be open to non-Sun content as well, if it relates to improving the download experience and helping our customers with any download questions or issues.

Kudos to Lori Holmes who got the site up and running and seeded the content with a few of the top questions from the SDLC FAQ. Next, I dutifully scanned the wiki manual and jumped in with my first contribution (including a shameless plug for my blog). I would love to add links to more expert resources around the web, so please let us know any suggestions.

I believe anyone who's a Sun employee has read/write privileges automatically, and your contributions are welcome. Those outside Sun will need to request edit permissions to contribute, but I'm sure we can figure out how to enable that if you'll let us know of your interest.

Time will tell if we can get critical mass around this effort, but at the least it allows us to quickly put out new information ourselves (without bugging our web publishing team to update the SDLC FAQ). Hopefully it will evolve into a valuable resource, ideally one with contributors far and wide!


Tuesday Feb 05, 2008

Breaking the Large File Barrier


"Large file" is actually a technical term for files larger than 2 GB (and to be really precise, that's 231 or 2,147,483,648 bytes.) In the past, we've not been able to distribute files over 2 GB on Sun Download Center (SDLC). Our engineers tell me that's because 32 bit systems cannot handle signed integers greater than 2 GB. Way back when we built our "old" download system, it didn't really seem to matter, as we would never try to offer files that large for downloading (kind of like not worrying about Y2K back in the 1980's!). But times have changed, and with the proliferation of large, single file DVD ISO images that many Linux distros use, it's no longer uncommon. (Of course, this goes hand-in-hand with the proliferation of broadband access.)

As we built our new download application, large file support was a requirement. But, we were still stuck with a large file limit in some of the older code in Sun Download Manager (SDM). As using a download manager is really, really helpful for files this large, we appeared to be unable to proceed. We were very aware of this limit and had a high priority mandate to fix it, but we haven't had enough engineering resources to take that on yet. The pressure was building, however, as the Solaris OS team really wanted to be able to release single large file DVD images (and I don't blame them).

That's when things got interesting. Internally, we were testing our new download system, and we put a few large files on it. One of the testers sent in some test results saying, "Successfully downloaded the 3.2 GB test file using SDM." Impossible, I thought, there must be a mistake. But no, the tester insisted it worked. So I tried it myself -- it worked! They say "ignorance is bliss," and thankfully this tester was unaware of what we all knew "would not work" and simply went for it. It was quite a surprise.

Now these types of bugs really don't have a habit of fixing themselves, but we figured out what's going on.

For files on SDLC, we generate what we call "Verification Property Files" (VPFs) that contain the checksums SDM uses to checksum downloads real-time as they are received. Another piece of data in VPFs is the file size, and that's how we get around this limit in SDM. It turns out that as long as there is a VPF for a large file (and we create them automatically for all files released on SDLC), SDM can get the file size from the VPF and it all works! When there is no VPF, the file size is part of the header info sent from the web server, and this is when things break. (Some older web servers can't handle the large numbers either.)

So, bottom line, after a bunch more testing, we've just released the first ever single large file on SDLC -- the latest version of Solaris Express Developer Edition (~ 3.7 GB DVD ISO image). This is a small but significant milestone after years of butting up against the 2 GB limit.

Now the larger the file, the more that can go wrong, so if you give this a try, please do use SDM. And here are some notes and "best practices" we gleaned from rolling this out:

  • The "32 bit" limit isn't unique to our systems but can affect servers, routers, operating systems, and clients throughout the network. For example, if a Windows XP system uses a FAT32 file system rather than NTFS, there's no way it's going to work -- the OS simply can't handle the file. (Thanks to openSUSE.org where I found that tip.)
    • As a result, we highly recommend to our product teams that they not rely solely on a large file for distribution, as it's not going to work for all customers. Offer options such as a "chunked" version of the DVD that users concatenate after downloading in smaller pieces. Or offer multiple CD images instead of the DVD (as we do for Solaris). And finally, offer a hard media version (DVD) that users can order inexpensively (or better yet, free) and is then shipped to them.
  • Use an up-to-date browser and fully patched, modern operating system to be sure large files are adequately supported on the client end.
  • Absolutely do not attempt this with a slow line, like a dial-up modem. You can expect it to take about 40 hours per GB on a 56K modem.
  • Use a download manager so you can resume where you left off in case anything goes wrong (you do not want to have to start over from the beginning). You can also pause and resume, if you're running out of time.
  • Make sure you have at least twice the size of the file in free disk space -- with these large files, that's actually quite a bit of disk space. Operating systems typically make a temporary copy of the file while downloading, then copy it to its final location, so you must have the extra space.
  • And a couple of notes specific to SDM:
    • As noted previously, SDM will not support large files except on SDLC, so don't try it on other sites (until we can get this fixed).
    • When SDM finishes downloading the large file, there is internal processing that must take place before the download is actually complete. Due to the huge file size, this processing can take several minutes. As a result, the SDM progress bar will say "100%" while the Status still says "Downloading data..." Be patient and do not close SDM. After a few minutes, the Status changes to "Downloaded", and the download is complete.

Hopefully this first large file release goes well and is the first of many. If you give it a try and have any problems or questions, please let us know -- the feedback is very helpful as we learn the ins and outs of large file distribution over the Internet.


Tuesday Dec 18, 2007

Sun's New Download System is rolling out now


After a few delays due to the complexity of the undertaking, I'm really happy to say we're back on track with the roll-out of the new download system that powers Sun Download Center. We are following a "phased product migration" plan which means we will not transfer all the products in one "big bang" from the old system to the new -- too much risk in that approach.

Back in October we put the first few products out but then had to hold pending some more back-end fixes. In early December, we released the fixes and then released 2000 of our lower volume downloads onto the new download system. A few days after that we released three high volume download products to slowly add load to the system. Among those is Sun Download Manager, which gets close to 30,000 downloads a month. So far so good -- all is working smoothly, and we have a ton of capacity remaining on our new servers.

Most of Sun closes for a week between Christmas and New Years, and so we decided to wait until after the break to move the remainder of the downloads onto the new system, including our "crown jewels" such as Solaris Operating System, Sun's Software PortfolioJava software and developer tools, etc. This ensures that we'll have the appropriate support and engineering staff available during the transition.

As you download from Sun for the next few weeks, you'll find some products on the old system and some on the new. Other than a few UI changes and other mostly-subtle improvements, this should be transparent and not an issue for any of our customers. (You can tell you're on the new system because the URLs start with https://cds.sun.com/...)

I'll be off shortly for a couple of weeks of much needed rest and relaxation so wanted to send my best wishes to you all for the holidays and a great 2008!


About

I helped design, build, and manage download systems at Sun for many years. Recently I've focused on web eMarketing systems. Occasionally, I write about other interests, such as holography and jazz guitar. Follow me on Twitter: http://twitter.com/garyzel

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
News

No bookmarks in folder

Blogroll
ESD