The Importance of Archiving (For the Rest of us)
By user13366078 on Sep 17, 2007
A few weeks ago I was on the road with Dave Cavena. He works as an SE in Hollywood and helps our customers there understand the importance of digitally archiving their movies. The issue here is a simple one: Today, movies are being archived by storing rolls of film on shelves in gigantic warehouses and hoping they'll survive for a few years to come. "Few" could be tens or maybe hundreds of years, but nobody really knows how long they'll really survive and how good the movies will look after a couple of decades of archiving. Will the colours look natural? Will there be scratches? Will the film material degrade so that the movie rips right in the middle of the most important scene? Or will it spontaneously decompose into a heap of dust when someone opens the door after 150 years to see what the heck people were keeping in that warehouse anyway?
Digital Data on the other hand can be kept indefinitely and it can stay perfect through eternity, if people store it right. "Right" means things like keeping redundant copies in geographically distant places (so that the movie survices a warehouse fire or an earthquake), periodical integrity checking and fault healing based on those redundant copies (so silent data corruption can be detected and corrected) and periodic copy and conversion cycles so the data can survive format and media evolution. Try playing "Dragon's Lair", a classic arcade game from the 80ies which was originally produced for Laserdisc-based arcade machines. I loved the game back them and I was glad I found it on DVD. Now look it up on Amazon: You'll find Blu-Ray and HD-DVD versions as well!
Dave and some other very bright people have written an interesting white paper on "Archiving Movies in a Digital World". It is a great read: It shows why archiving movies the digital way is so important (so they can't get lost), how to do it and why this is actually cheaper than keeping rolls of film in warehouses (Hint: Archiving bits takes up less real estate and looks a lot cooler if you use one of these. Your archives may even become smart by using one of these, too!).
That got me thinking: What will happen to all those photos that people are taking using their private digital camera? Parties, Vacations, Babies, Families, etc.? Yes, if people don't start thinking about a good archiving strategy soon, they will all be lost in the next couple of decades. By the time my little daughter gets married, I might have lost all of her baby pictures if I don't do something real quick (as in: This decade).
Storing them on a file server running a serious, enterprise-class OS with ZFS on a set of redundant storage media (could be disks, could be more disks, iSCSI devices, could be USB sticks, it really doesn't matter) is a good start, because it can provide 100% data integrity and self-heal any damages before they become permanent. But this is still not enough. They need to be stored in multiple locations and they need to be periodically copied into more recent media.
I'll figure out the multiple locations part someday (probably a second server that replicates the first file server's ZFS file systems through a couple of zfs send/receive scripts) and the periodical copying means I'll still have an interesting hobby when my daughter has children of her own. Meanwhile, just to be sure, I've started to copy all of my photos to a popular photo-sharing service called Flickr on the net. Yes, all of them. This means I can still decide who can look at my pictures and who not, I get access to all of my pictures from everywhere near a web browser and I can store as many photos as I want for just a small annual fee. And they'll still be there should my basement catch fire or should all of my disks die for some strange statistical reason or when I start taking 3D photographs of my grandchidren. What more could anybody want?
Now what if the movie industry found out about this and started archiving their movies on Flickr as well, image after image, all of them, for just the same small annual fee?
Update (Sep. 18th, 2007): Thanks to Jesse's comment, I've now tried SmugMug and I love it! They offer a 50% discount for Flickr refugees during the first year and there's a nice tool called SmuggLr that makes migration a snap. Thank you Jesse!