A Brief History of Sun Download Manager
By Gary Zellerbach on Jun 16, 2006
I've been charged with leading the project to develop the "next-generation" of Sun Download Manager (SDM), which we'll dub version 2.0. I gladly take this on, as I was part of the team that developed the first SDM and feel a strong kinship. Developing the next version is going to be a very interesting and challenging project, as we've some tough business requirements to meet while maintaining a 100% Java application. I'll get into some of the details and goals in the future, but for now I wanted to capture a bit of SDM history.
It all started over 5 years ago when we began distributing the Solaris Operating System at no charge over the Internet. This proved to be a very popular move, but also quickly proved the point that downloads are not free to those who host them. When our download team presented a bandwidth bill to the Solaris team that was, shall we say, not inconsequential, things got "interesting" very quickly. The kicker was when our analysis showed a very low success rate (under 20%) for the downloads. At that time there was very little broadband and 300 MB Solaris files were really quite large downloads. It was reasoned that we could save a lot of money if the success rate were much higher -- we wouldn't have to pay for all that wasted bandwidth so the same people could download over and over again until they (hopefully) succeeded.
A Sun Sigma team was formed to do what Sun Sigma teams are supposed to do -- perform an in-depth analysis of a problem by gathering and analyzing data, then come up with a solution. We realized we could greatly increase the success rate with some easy, "low hanging fruit" changes. These consisted primarily of educating customers up-front about what it would take to have a successful experience when downloading such large files: allow yourself plenty of time, make sure you have adequate disk space (2 to 3 times the size of the files being downloaded), use a download manager, and don't bother unless you have a decent Internet connection (buy the CD instead!). We added these instructions (along with great info on what to do after downloading, such as how to burn the CDs) as web pages users had to traverse to get to the download links. The reading and extra clicks may have turned a few people off, but it sure improved the download completion rates. These simple changes easily doubled the rates.
That was good, but not exactly Six Sigma completion rates (less than 3.4 defects per million opportunities)!
So our work was only half done. To reach our real goal of 100% success rates, we realized we needed to provide a tool to our customers that would insure a successful download, and thus the development of SDM began. Why our own download manager when there were plenty available? The main thing is we wanted to calculate checksums on every download, as that seemed like the only way to insure 100% success. Some tools offered this, but checksums were run after the download completed. If the checksum failed, the entire 300+ MB file would need to be downloaded again, and there went our cost savings.
We invented a method to run checksums on every single MB of the file as it was downloaded. If a 1 MB byte range failed a checksum (calculated just for that range), a byte range request was made to the server for just that byte range. With this method, customers who used SDM achieved virtually 100% success (as long as they would complete their download), and everybody saved on the time and bandwidth usage.
This solution has served us very well. Completion rates are much higher now (and of course the infrastructure has improved tremendously too), and SDM continues to be one of our most popular downloads every month. (As it's a 100% Java application, it runs on any platform with a current JVM, so we know it's not just used on SDLC. Many people download it for general use on Solaris systems as well as Linux, Windows, etc.)
That brings us to the present. Those who use SDM have great success, and we all benefit from successful downloads and more satisfied customers. But not everyone uses SDM, and that's our challenge. We need to make it easier to get and use SDM and integrate it directly into the download process. I'll give some hints how we're going to do that in a future post.