Why Downloads Fail (Part 2)
By Gary Zellerbach on Nov 17, 2005
In Part 1, I defined some key terms and concepts about measuring download success rate. Now I'm ready to drill down into some details.
So, why do downloads fail?
Note that I'm using the word "fail" to mean "did not complete for whatever reason." I do not mean to limit the discussion to technical failures. It's especially important to make that distinction because, in fact, we believe the number one reason is not technical -- the main reason seems to be users stop the download.
From the server side, we only know the stream of bits going to the customer was broken before the last bytes were served. We really can't tell if that was because something outside our control "broke" (technical issue) or the user simply hit "cancel". We suspected user actions played a large role but weren't sure how to "prove" this. While I'm not saying we did prove this 100% scientifically, here's some of our reasoning for the hypothesis provided.
First, while our infrastructure provides all the downloads on Sun's download center, the same infrastructure also supplies all the files for Java SE and JRE auto-update functionality. We noticed the completion rate for similarly sized files was higher for auto-update. Since the files are on the same infrastructure and of similar size, what would explain the difference? A plausible explanation is that once a user approves the auto-update, the downloading happens automatically and with no readily apparent way for a user to stop the download once it's started. This would seem to indicate that the more automated you make a download, and the less chance a customer has to cancel it, the higher the success rate.
Next, we had a really large file (400+ MB) with a success rate of under 50%. Similarly sized files were running 65-75%, which really is very good for such huge files (based on our experience as well as some benchmarking with other vendors). The product team with this particular file kept suggesting there might be something wrong with our system, so we tried to dispel that. We took two specific actions:
- We set up a large file synthetic transaction (LFST). A synthetic
transaction is an automated, scripted web transaction that mimics
exactly the steps a user takes to download a file. We put the same
problematic file on public-facing servers in the US, Holland, and
Singapore. Our LFST then ran every 3 hours 24x7 against each location
-- it logged in to the site, accepted the license, and downloaded the
file, exactly as a "real" human would do. The results were
enlightening. The success rate was greater than 99% in the US and
varied (over time) in the other locations from a low in the mid 80's to
an average in the high 90s. These downloads traveled over the Internet
to these locations from the download center, just as if end customers
were downloading. So if a "robot" gets 85%-100% success rate but the
public achieves 50%, isn't the most likely explanation that user's
abort the download? (In some cases, we know users may have
infrastructure issues not associated with our test sites, but frankly
our sites weren't that well-connected outside the US, using generally
highly-taxed Internet pipes out of smaller data centers.)
- For further confirmation, we sent a survey to customers who had
downloaded (or tried to download) this product and asked them about
their experience. Note surveys can skew toward those who are
dissatisfied and want to tell you about it, but even with that taken
into account, there were very few problems reported. Most users said
they got the file no problem. If technical issues were
contributing to a 50+% failure rate, I think we would've heard it loud
and clear. Instead, customers were overwhelmingly "satisfied" or "very
satisfied" with their download experience.
Of course there are technical problems that cause downloads to fail (I'll get to some of those in "part 3"), and "blaming" our customers is not a solution. But these findings seem to validate that the main reason downloads fail (for us, at least) is that users cancel their downloads before they complete. While the information is valuable, I'm certainly not saying this is a good thing! We need to continue to understand why this happens and how to mitigate it, as we're always trying to improve completion rates and customer satisfaction. (And I'll get into some ideas on how to do that as well at a later date). But we do believe these findings indicate we don't have any systemic technical issues with our download infrastructures and applications, and that is a good thing.