Wednesday May 27, 2009

Deploying Sun's Latest "Advanced Download Widget"

I first wrote about our forays into "1-click" downloading a while ago (could it really be two years already?), and as that post noted, there were significant limitations in our first 1-click widget. We had to take on some higher priority projects that delayed our ability to address the limitations for a while, but as you'll see, we're now solidly back on track. 

While I don't work full-time on downloads any more, I am still the Product Manager of the Sun Download Manager (SDM) and of course remain very well-connected with the Download Team. As such, I was asked by Alfred Chen, the current Download Team Lead, if we could try out the latest version of the "advanced download widget" (ADW) on the SDM download page. SDM remains one of our most popular downloads so would get plenty of traffic for "real world" experience and also met the criteria for the new ADW. I was glad to say "Yes," and we went live with it several weeks ago. 

One of the main limitations of the original 1-click release was that it worked for one file and one file only. The new ADW now accommodates multiple platform and language options so is much more versatile.  It even "auto-detects" your platform and language and will pre-select the appropriate option when available. Once platform (and if available, language) are selected, it automatically pushes the correct binary for 1-click downloading. Here's a screen shot of the new widget:

Sun Download Manager Advanced Download Widget

You can try it out for yourself by downloading SDM.

There's more work to do, however -- ADW still does not support optional or required registration, so that's the next set of enhancements. They are underway and if all goes well will be released this summer. I have previously touted the virtues of optional registration, and there are several places on Sun now where we're using this with very good results. For example, on the Glassfish Download Page, there is an optional registration "pop-in" widget that comes up when you click the Download button. This illustrates a new twist on "optional download registration," as it includes the added incentive of receiving an informative white paper for registering. Being optional, you can just close this out and the download will start automatically (there's that 1-click functionality again!), but many customers are taking advantage of the white paper offer. So they get a white paper and the Glassfish download, and we get valuable customer information. 

Glassfish optional reg component

While this new model is working great, the Glassfish implementation is frankly a "one-off". What the next version of ADW will provide is a standard way of doing this across Sun sites with minimal effort. I'm really looking forward to this new functionality and think it'll be a great win for both Sun and our customers.  It continues the march towards our goal of seamlessly integrating the download experience directly into the web pages you're on, rather than forcing customers to "move" between the web site and the "download system." 

Friday Oct 24, 2008

Download Best Practices, Part 3

Here's a "best practice" that is a source of never-ending debate: Should we require login/registration in order to download our products? There is no right answer, but there are plenty of pros and cons.

Sometimes, we have no choice but to require login in order to comply with government export regulations. (For example, if a product contains some types of strong encryption, we must keep a record of who has obtained it.) An obvious reason in favor is that knowing who downloads our software is of great value in follow up marketing initiatives as well as demographic analysis of our customer base. It's not all a one-sided equation though. For our customers who login first, we offer a useful download history function, and we also automatically provision our My Sun Connection portal with a list of products of interest to you. (This automatic portal provisioning is set up for many, though not all, of our downloads.) It's very handy, as you can click on product names and quickly see a valuable list of related resources and information. (I encourage you to check out the portal if you're not familiar with it. Once there, be sure to click the "My Products" tab to see how we aggregate product information from many resources in one place!)

The other side of the coin is that requiring login or registration before downloading definitely hurts the conversion rate. Using our web analytics, you can see users drop off with every extra click, and the largest abandonment is at the login/registration step. In many cases, the value of getting our software into the hands of our (often first-time) customers as quickly and with as few roadblocks as possible trumps the value of collecting user information, so we do not require registration. 

With such conflicting agendas, you must be wondering by now what I'm going to recommend as the best practice? Well, that's easy, don't use either -- use optional registration! This feature provides a middle ground that works for most products -- just give customers the choice. If they want to login to gain the benefits offered, we welcome and appreciate it and believe it's worthwhile. And if the customer just wants the software as quickly as possible, there's no need to register, just click Continue. Here's what you'll see when you request an "opt reg" product on Sun Download Center:

optional registration interface

For those who do choose to login first, we find a better quality of data since they are doing so of their own volition, and it allows us to build a stronger relationship with the customer. Of course, we have to face the facts -- a majority of customers will skip the login step and click Continue. For that reason, we highly recommend building optional product registration into the product's installer as well. Once you have committed to installing the product, there's an even stronger opportunity to build the relationship and help support the software on your system.

Fortunately we built our download system to support all three options (required, optional, none) easily, so we can meet whatever the product team wants with the flip of a bit in the product database. However, our recommendation remains to use optional registration whenever possible.

This is the third in a series of download best practice posts.

Friday Oct 03, 2008

Two new bloggers now writing about downloads


After more than 10 years of focus on download systems here at Sun, I was ready for a change of pace (as you might imagine). Once we got our new download system out the door, my manager was much more willing to consider a change for me, and then a great opportunity arose. I'm now managing a new project that is developing a machine learning engine for better targeting our offers to our customers. While the obvious intent is to benefit Sun by increasing the uptake on our products and services, we feel it benefits customers as well by providing more relevance to the ads and content we provide you on our web sites.

It's pretty much a whole new world for me, but I'm enjoying the challenge and learning about a whole new area of analytics. It would be a shame to ignore the experience I've acquired in the ESD realm, so I'm still participating in download strategy discussions and helping out where I can. I may well blog further on the subject as time allows.

But in any case, I am happy to see two new blogs about downloads and ESD that have sprung up recently. As I wrote in my very first blog posting three years ago, "I haven't found many resources for us 'ESD Professionals.' I'm not aware of trade journals nor web sites dedicated to ESD. I always enjoy meeting and speaking with my peers at Sun and other companies but wish there was more interaction, more sharing of best practices, and the myriad other benefits that come from knowledge sharing in a particular field. This blog is intended to be a small outreach..."

Well, I can't say I've seen an explosion in content, but I heartily welcome two new contributors to the ESD blogosphere!

  • The overall Project Manager of the new download system was Alfred Chen, and he's the guy in charge now that I've stepped into a new role. Alfred is leading our Download Business and Strategy Teams, and we are very lucky to have him in those roles. Working with Alfred very closely for many years, I can only say he's a tireless contributor with exceptional knowledge, skill, and dedication. (And he used to be the manager of our download engineering team, so he's really one of those great persons with deep IT and Business experience.) So, to get to the point, Alfred's started blogging lately (about a variety of subjects) -- check it out.
  • OMS Safe Harbor is one of the few companies out there with a major focus on electronic software delivery, and they've built up an impressive set of capabilities, products, and services. They recently let me know about their new ESD blog, and I've already read a number of interesting articles there. Great to see this outreach from OMS.
I hope you'll find these new bloggers of interest!


Tuesday Jul 08, 2008

A Standard for Measuring and Reporting Downloads


Mozilla and Firefox have garnered a lot of well-deserved publicity for their new Guinness record for most downloads in a day. This raises the question (again) of what constitutes a "download" and how is it measured? (I first touched on this subject a couple of years ago, writing "When is a download a download?".) I was glad to see in the above referenced BBC article, "The official figure was confirmed after logs from download servers were audited and checked to ensure duplicate and unfinished downloads were not counted." So that implies "attempted" downloads weren't counted, only completed. That's good, as I'm sure some stats we see reported don't make such distinctions. 

So with Guiness in the mix now, and with all the focus on downloads in the free & open source software world, I propose it's time for an agreed upon standard on how the number of downloads is reported. (To be honest, I'm uncertain if this could ever be a "formal" standard and who would govern it -- please let me know if you have any suggestions.) For starters, here's what I propose:

  • Simple case: One Product = one file
    • This case means that the entire product is contained in a single downloadable file. If it is downloaded successfully, the customer can install and use the product.
    • The suggested measurement is the one generally used -- the last byte of the file is delivered to the customer.
      • Note this may not be 100% accurate. Threaded download managers can conceivably download the last byte of a file and then fail an earlier byte range before the entire download completes. But this is an edge case. It would also be extremely resource intensive to have to match up all the byte range requests in a threaded download scenario to "prove" that one user completed the download. Thus, I think the "de facto" standard of last byte delivered is acceptable.

  • Complex case: One Product = multiple files
    • First, make the distinction between required and optional files:
      • Required files are required to install the product. There may be one or more required files as part of the download. 
        • For example, on Sun Download Center, there is an option to download Solaris OS as 5 CDs. If you do not download all five, you cannot install the product. Thus, all five are required files, and each one must be downloaded to install Solaris.
      • Optional files are not required to install or run the product. Optional files may consist of "plug-ins,"  "add-ons," or "modules" that add functionality to the base program. Also included would be documentation, checksum files, additional language packages, and the like.
    • The measurement of a completed download in this case is that all required files are downloaded by the same customer. (This can be a challenge to measure, but it can be done in most cases with a good degree of accuracy.)
    • Optional files do not count and should never be reported as "product" downloads.

  • In either case, "Attempted Downloads" should never be reported as a product download. 
    • For a single file product, an attempted download is when the user starts the download but does not receive the last byte of the file.
    • For complex products, an attempt is when any individual required file is not completed and/or the customer does not successfully download all required files.

Monday Jun 02, 2008

Download Best Practices, Part 2


I'm going to list three more of the download best practices that we encourage all our product teams to use as they develop products for electronic software distribution (ESD). This is the second part of a continuing series I'll publish from time to time. (You can see Part 1 here.) Some of these suggestions may seem like "no-brainers" (at least once you understand the basics of ESD), but they need to be stated and documented nonetheless and are certainly helpful to teams that are just starting out to prepare downloadable software.

1) Compress the files as much as possible. Obviously smaller files are cheaper/faster for everyone to download, and faster delivery means less time for things to fail and higher completion rates. Generally, we recommend using zip compression, as it's pretty much a de facto standard at this point (and is supported on Solaris OS as well). There are many free and open source distributions product teams can use, such as Info-ZIP, bzip2, and 7-Zip. You should experiment to see which one provides the best compression (in one test here, 7-Zip outperformed the others by about 20%, but your results may vary). There are also proprietary compression programs ("NOSSO," for example) that can greatly outperform "vanilla" zip compression, but they come with a price tag. If your download size really really matters, they may be worth investigating.

2) Use standard MIME types for file name extensions, for example "filename.zip" and "filename.exe." Web servers, browsers, download managers, and many other pieces of the Internet infrastructure rely on standard MIME types to ensure download transactions work correctly and consistently. So, don't release files without a MIME type extension (for ex., use "README.txt," not "README"), and don't make up file extensions or use non-standard ones, such as "filename.aa," "filename.bb," etc.

3) For large files, offer alternatives. Not everyone can successfully download a large file, especially ones larger than 2 GB. Other choices should be provided, such as a segmented ("chunked") version of the large file, CD images instead of a single DVD, and/or the option to order free or low cost CDs or DVDs. See my earlier post on "Breaking the Large File Barrier" for more details.


Thursday May 15, 2008

Using wget With Our New Download System


Soon after releasing our new download system, we received comments that the command line download client wget (and others) no longer worked. This is a result of a new session-based security model used to protect download links from being copied and re-used (which could circumvent numerous Legal requirements necessary to authorize a download). 

We have implemented a work-around now, so I just wanted to put this quick note out there to increase awareness of its availability. If you'd like to use wget or similar tools with Sun Download Center, please review the work-around, posted on our new Downloads FAQ Wiki.

Sorry for any inconvenience this has caused -- it was one of those unintended consequences. We believe we have a solution for it and will implement it as soon as we can. In the meantime, I'm pleased we could at least offer a temporary fix.

Get the work-around script here: Using wget with the Sun Download Center


Friday Apr 18, 2008

Video Demo of Sun's New Download System


OK, so things took a bit longer than we expected, and we hit a few bumps in the road, but our new download system is now fully operational and humming along! Just this week we finished moving all sun.com downloads onto it, and we're extremely pleased to reach this milestone.

To help introduce the new system (and also get newcomers and experienced users alike downloading as quickly, easily, and successfully as possible), please check out our new demo video. It features a walk through of the Solaris OS download experience, as well as helpful tips on using Sun Download Manager. The video takes less than five minutes.




I hope you enjoy the video, and more importantly, that you benefit from the enhancements we've made to downloading from Sun. As always, your suggestions for further improvement are welcome. You can leave me a comment, or if you encounter any questions or issues, please let our customer support team know. Thanks!


Friday Mar 21, 2008

Download Best Practices, Part 1


We've been working on a new internal wiki to capture download best practices gleaned over the past 11 years of designing and managing our high capacity download systems. I think this is good and valuable information so wanted to blog about some of our findings. This will be the first in a series of posts on the subject.

First, let's consider our goals in defining and publishing these practices. At a high level, I'd sum them up as follows:

  • Ensure customers are able to download successfully, as fast as possible and with minimal hassle.
  • Provide a consistent and efficient download experience.
  • Increase completion rates and reduce abandonment.
  • Ensure our systems and processes optimize the conversion of downloaders into (paying) customers.
  • Save time and bandwidth (cost) for Sun and our customers.
  • Within Sun, document the processes to release software for download, and ensure they are followed.

Our first best practice is seemingly simple yet frankly it's not been followed consistently over the years: use a unique file name for every file in every product and version release. There are several reasons why this is so important, the first of which is unique to our systems:

Sun Download Manager uses special "verification property files" (VPFs) to checksum files as they are downloaded from Sun Download Center. The VPFs are mapped to unique file names. So, here's where we get into really deep water if file names aren't unique from release to release! If we release a product with file name "product.zip", a VPF will be created called "product.zip.sdm.zip" that SDM grabs automatically and uses to checksum the file and ensure a successful download. Now if the product team releases the next version of the product but does not update the file name, a new VPF will be created with the same name as the prior one. This works fine for downloading the new product. But when users try to download the prior version, there will be a checksum mismatch between the old file and the new VPF -- the download will fail. And that's why this is so critical on our system.

Here are a couple of other good reasons (not unique to Sun's download systems):

  • Downloads often flow through content distribution networks which cache files around the world for fastest delivery. Unique file names ensure caching works most efficiently and eliminates the potential for delivering incorrect content from the caches.
  • Unique file names reduce potential errors when locating, copying, moving, updating, reporting on, and archiving content.

It's really not hard to do, so we hope you'll follow this practice!

If you work for Sun, you can view the full Download Best Practices Wiki.

Sorry, this is for Sun employees only. The link will not work unless you are inside Sun's firewall.


Thursday Feb 14, 2008

New Wiki for Sun Download FAQs


I've been blogging for a while now, and I guess it was inevitable that wikis were in my near future as well! Having spent a lot of time and energy learning HTML and web publishing, I was hesitant to dive in and learn "wiki language" and the new publishing method. But a little encouragement from our VP goes a long way -- Curtis suggested we take a stab at a wiki to involve more of our community in the areas of download support and information.

So, we've taken the plunge and have posted a new Sun Downloads FAQ on wikis.sun.com. We already have a comprehensive FAQ devoted to Sun Download Center, and there's no point in repeating all the content in a wiki format. Rather, we want to broaden the scope to include all Sun downloads (open source products, for example) and related areas. I think we should be open to non-Sun content as well, if it relates to improving the download experience and helping our customers with any download questions or issues.

Kudos to Lori Holmes who got the site up and running and seeded the content with a few of the top questions from the SDLC FAQ. Next, I dutifully scanned the wiki manual and jumped in with my first contribution (including a shameless plug for my blog). I would love to add links to more expert resources around the web, so please let us know any suggestions.

I believe anyone who's a Sun employee has read/write privileges automatically, and your contributions are welcome. Those outside Sun will need to request edit permissions to contribute, but I'm sure we can figure out how to enable that if you'll let us know of your interest.

Time will tell if we can get critical mass around this effort, but at the least it allows us to quickly put out new information ourselves (without bugging our web publishing team to update the SDLC FAQ). Hopefully it will evolve into a valuable resource, ideally one with contributors far and wide!


Tuesday Feb 05, 2008

Breaking the Large File Barrier


"Large file" is actually a technical term for files larger than 2 GB (and to be really precise, that's 231 or 2,147,483,648 bytes.) In the past, we've not been able to distribute files over 2 GB on Sun Download Center (SDLC). Our engineers tell me that's because 32 bit systems cannot handle signed integers greater than 2 GB. Way back when we built our "old" download system, it didn't really seem to matter, as we would never try to offer files that large for downloading (kind of like not worrying about Y2K back in the 1980's!). But times have changed, and with the proliferation of large, single file DVD ISO images that many Linux distros use, it's no longer uncommon. (Of course, this goes hand-in-hand with the proliferation of broadband access.)

As we built our new download application, large file support was a requirement. But, we were still stuck with a large file limit in some of the older code in Sun Download Manager (SDM). As using a download manager is really, really helpful for files this large, we appeared to be unable to proceed. We were very aware of this limit and had a high priority mandate to fix it, but we haven't had enough engineering resources to take that on yet. The pressure was building, however, as the Solaris OS team really wanted to be able to release single large file DVD images (and I don't blame them).

That's when things got interesting. Internally, we were testing our new download system, and we put a few large files on it. One of the testers sent in some test results saying, "Successfully downloaded the 3.2 GB test file using SDM." Impossible, I thought, there must be a mistake. But no, the tester insisted it worked. So I tried it myself -- it worked! They say "ignorance is bliss," and thankfully this tester was unaware of what we all knew "would not work" and simply went for it. It was quite a surprise.

Now these types of bugs really don't have a habit of fixing themselves, but we figured out what's going on.

For files on SDLC, we generate what we call "Verification Property Files" (VPFs) that contain the checksums SDM uses to checksum downloads real-time as they are received. Another piece of data in VPFs is the file size, and that's how we get around this limit in SDM. It turns out that as long as there is a VPF for a large file (and we create them automatically for all files released on SDLC), SDM can get the file size from the VPF and it all works! When there is no VPF, the file size is part of the header info sent from the web server, and this is when things break. (Some older web servers can't handle the large numbers either.)

So, bottom line, after a bunch more testing, we've just released the first ever single large file on SDLC -- the latest version of Solaris Express Developer Edition (~ 3.7 GB DVD ISO image). This is a small but significant milestone after years of butting up against the 2 GB limit.

Now the larger the file, the more that can go wrong, so if you give this a try, please do use SDM. And here are some notes and "best practices" we gleaned from rolling this out:

  • The "32 bit" limit isn't unique to our systems but can affect servers, routers, operating systems, and clients throughout the network. For example, if a Windows XP system uses a FAT32 file system rather than NTFS, there's no way it's going to work -- the OS simply can't handle the file. (Thanks to openSUSE.org where I found that tip.)
    • As a result, we highly recommend to our product teams that they not rely solely on a large file for distribution, as it's not going to work for all customers. Offer options such as a "chunked" version of the DVD that users concatenate after downloading in smaller pieces. Or offer multiple CD images instead of the DVD (as we do for Solaris). And finally, offer a hard media version (DVD) that users can order inexpensively (or better yet, free) and is then shipped to them.
  • Use an up-to-date browser and fully patched, modern operating system to be sure large files are adequately supported on the client end.
  • Absolutely do not attempt this with a slow line, like a dial-up modem. You can expect it to take about 40 hours per GB on a 56K modem.
  • Use a download manager so you can resume where you left off in case anything goes wrong (you do not want to have to start over from the beginning). You can also pause and resume, if you're running out of time.
  • Make sure you have at least twice the size of the file in free disk space -- with these large files, that's actually quite a bit of disk space. Operating systems typically make a temporary copy of the file while downloading, then copy it to its final location, so you must have the extra space.
  • And a couple of notes specific to SDM:
    • As noted previously, SDM will not support large files except on SDLC, so don't try it on other sites (until we can get this fixed).
    • When SDM finishes downloading the large file, there is internal processing that must take place before the download is actually complete. Due to the huge file size, this processing can take several minutes. As a result, the SDM progress bar will say "100%" while the Status still says "Downloading data..." Be patient and do not close SDM. After a few minutes, the Status changes to "Downloaded", and the download is complete.

Hopefully this first large file release goes well and is the first of many. If you give it a try and have any problems or questions, please let us know -- the feedback is very helpful as we learn the ins and outs of large file distribution over the Internet.


Tuesday Dec 18, 2007

Sun's New Download System is rolling out now


After a few delays due to the complexity of the undertaking, I'm really happy to say we're back on track with the roll-out of the new download system that powers Sun Download Center. We are following a "phased product migration" plan which means we will not transfer all the products in one "big bang" from the old system to the new -- too much risk in that approach.

Back in October we put the first few products out but then had to hold pending some more back-end fixes. In early December, we released the fixes and then released 2000 of our lower volume downloads onto the new download system. A few days after that we released three high volume download products to slowly add load to the system. Among those is Sun Download Manager, which gets close to 30,000 downloads a month. So far so good -- all is working smoothly, and we have a ton of capacity remaining on our new servers.

Most of Sun closes for a week between Christmas and New Years, and so we decided to wait until after the break to move the remainder of the downloads onto the new system, including our "crown jewels" such as Solaris Operating System, Sun's Software PortfolioJava software and developer tools, etc. This ensures that we'll have the appropriate support and engineering staff available during the transition.

As you download from Sun for the next few weeks, you'll find some products on the old system and some on the new. Other than a few UI changes and other mostly-subtle improvements, this should be transparent and not an issue for any of our customers. (You can tell you're on the new system because the URLs start with https://cds.sun.com/...)

I'll be off shortly for a couple of weeks of much needed rest and relaxation so wanted to send my best wishes to you all for the holidays and a great 2008!


Tuesday Nov 27, 2007

Metalink takes off!


In a world of unintended consequences, one I often think about is not realizing how many new friends I would make because of my kids. Through numerous events, car-pooling, baby sitting, play dates, parties, and years of schooling with the same kids in their classes, I gained a whole new set of good friends (the parents that is, not the kids!).

Similarly, I hadn't realized when starting my blog that you can make some very interesting connections and virtually meet people who share your interests. It's a great benefit, and here's a great example. Soon after starting my blog, I "met" Anthony Bryan via some thoughtful, intelligent comments he left on subjects I was discussing. He must've found me due to my interest in ESD and download managers -- if you look around, there's simply not that much written about those subjects. And it's always great to find others who share your interests and passions.

I first mentioned Anthony's project, Metalink, almost two years ago, when he was just starting to gain traction. We've kept in touch, and it's really amazing to see how it's taken off since then. It's no accident of course -- it took perseverance, in combination with his clever, well-implemented, open technology. Metalink filled a gap in download managers and systems, providing for much needed enhanced redundancy, load sharing, and fault tolerance for large file downloads.

There's a long list of products now that incorporate Metalink, a sure sign of growing acceptance and success. I was going to mention a few, but I see his home page is up-to-date and says it much better than I can, so take a look. Also, here's an informative interview with Anthony about his project and its benefits.

So, what about Sun Download Manager (SDM), does it use Metalink? Well, no, not yet at least. The main reason is that SDM's primary audience is customers downloading Sun software from Sun Download Center (SDLC). Access to this software is carefully controlled for security and export control reasons. We use load management to distribute the load on multiple servers in our own data centers. As there aren't mirrors our there for this class of software (i.e., mostly not Sun's open source software), we lose one of the main advantages of Metalink. That said, we do know a lot of people use SDM on other sites because it's a good, simple, free, cross-platform download manager. So that sounds like a good argument to build in Metalink support in the future! (I'll say it before Anthony does.) I'll certainly keep it on the radar, but must admit all our engineers are tied up finishing our new download system at the moment.

In the meantime, I see that Metalink is used for OpenOffice distribution and was further pleased to see it mentioned in a number of other Sun blogs

Congratulations Anthony, and I hope Metalink is just the first of many successes for you. 


About

I helped design, build, and manage download systems at Sun for many years. Recently I've focused on web eMarketing systems. Occasionally, I write about other interests, such as holography and jazz guitar. Follow me on Twitter: http://twitter.com/garyzel

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
News

No bookmarks in folder

Blogroll
ESD