Wired: One Giant Screwup for Mankind

Several weeks ago I blogged about data loss and taking the long view when it comes to data retention. This month's Wired magazine has an article entitled One Giant Screwup for Mankind that illustrates the need for taking a long view of data retention policies. It also brings up an interesting point about our current trend at digitizing and chopping our digital content up into lossy data compression formats (like 128kbps MP3s).

Apparently, the grainy images of the original moon landing that we see on TV ("one small step for [a] man...") are not the original images and sound! The engineers were forced to create a smaller format for transmission from the moon to earth, it was 320 scan lines at 10 frames per second transmitted at 500 kHz. This stream was received at 3 tracking stations, pushed to a central location, recorded on media and converted to the broadcast rate of 525 scan lines at 30 frames per second transmitted at 4.5 Mhz. This is essentially 3 transmissions (camera to tracking station, tracking station to central site, central site to tv) and 2 conversions (camera to moon/earth broadcast, moon/earth broadcast to tv). Between the reception of the data and conversion to the tv format, the quality was greatly reduced! The engineers noticed that the broadcast images were not as crisp as what SHOULD have been in the original format. In fact, they could verify this with pictures of the monitors in the conversion room. So, the engineers tried to find the tape that the original data was recorded on so they could recover the full quality images.

Gone, lost, disappeared.

Just as I mentioned previously though, the engineers had TWO problems they had to work on:
- Getting and retaining the equipment they could use to recover the origninal data (remember, we have went through multiple media formats since the 60s)
- Locating the original tapes used for recording the data stream prior to conversion to the television signal format

I won't tell you how its going, you have to read Wired to find out. But, this does bring up an excellent example of
- Why a company that has record retention requirements of over 7 years must put in place a comprehensive policy to not only record the information and store it, but also retain the equipment that can read that data and write it to a new format. Some companies, instead of storing the components to read/write data, will enact a policy to migrate the data to the current media format every 7 years or less.
- Why a company should consider the effect on history of losing their data if their retention policy is less than 7 years or not explicitly stated. For example, is there a retention policy at our record companies for all of the garage band tape recordings they've received? If there isn't, how are we going to retain this valuable piece of American History and Culture? The record companies have a historical responsibility to record and maintain these.

More interestingly to me for this blog post is the problems with the data conversion process itself. Recall I'm a big vinyl fan at this point. Vinyl and analog recordings provide a warm and continuous signal whereas digital chops that up into many slices. Further, when compressing information for our MP3s we actually lose data. Depending on the number of Kbps you use, the data loss can be very noticeable in certain types of music.

Many download services also do not provide lossless downloads.

In the coming year we will see 1 Terrabyte desktop drives. I am convinced that we will start seeing more pervasive use of lossless compression. Still, it begs the question, will our original data remain intact? Are we losing important historical data and content quality through the conversion to digital and then using lossy compression techniques because we feel the quality is "good enough"? I have every reason to believe that as we start merging technology with our bodies and brains, our senses will become more and more aware of the lossy compression techniques used in the late 90's and early 2000's. Even without computer enhancement our brains are adapting to the saturation of media and information in a way that previous generations would be astounded at.

The only question to our kids who will have the heightened senses through the merging of technology with our human anatomy will be "How much quality did my parents compromise and lose for the sake of their convenience...and how much of it will we be able to recover to enjoy their creativity to its fullest potential?". So, be sure to save those original recordings...especially if you are the owner of the Beatles recordings.

btw, does anyone REALLY agree with releasing a Beatles album that does not adhere to the group's original music scores but is instead a mashup? Should content created by a team of people in a specific way be rebuilt to fulfill someone else's vision? What if our future generation actually thinks that these songs were originally mashed up, are we changing history? I agree with mashups and especially for content that is INTENDED to be mashed up, but I believe we should be very careful with taking original content and mashing it up to be something not intended by the author (though I do like the version of the Elvis tune at the beginning of the NBC show Las Vegas :-)

- Gotta run!


Interesting blog entry Paul.

However, I do have one counter-point, and that is at what digital fidelity do we no longer need analog originals?

For an audio example, average human hearing is generally defined as 20-20kHz, so any recording outside of this frequency range is wasteful with data. Likewise, if we record outside of this range and then strip the recording [compress] and only save the useful 20-20kHz portion, we are not losing anything ultimately useful for the intended purpose…no "data" loss.

From my testing, I cannot distinguish any fidelity difference between an original CD audio recording and a 192-kb/s Vorbis format, granted my original CD format was uncompressed digital, but let’s ignore that for now. Vorbis audio recordings at 128-kb/s exhibited slightly perceptible degradation from the originals with specific music content, but with a 160-kb/s Vorbis recording I could only rarely discern any difference from the original.

In conclusion, I would argue the definition of "data loss" is very important to discern, but more importantly to me, bit rot is of much greater concern given our growing appetite for digital storage. Thankfully, ZFS will help with this to some extent.

Posted by Wes W. on January 12, 2007 at 04:58 AM MST #

I agree, and disagree with you. I can't really distinguish between vinyl and CD. I can distinguish between CD and 128bit and with certain types of music I can distinguish between CD and 192bit, but the differences are becoming less. I now record in 256 and will likely switch to this variable bit rate I've been hearing about lately since I am getting more storage in my house. Still, my point was less about human hearing than enhanced human hearing or even "computer" analysis of music. We know for a fact that teenagers can hear tones that adults cannot (thus that crazy ringtone that only teens can hear). To some people, getting rid of that data would be acceptable, but to them...they would lose their ringtone to some compression algorithms :-) So we know that different people hear different frequencies and can distinguish different qualities so losing any data may be too much. You could argue that reducing the frequency is OK but my argument is less about today than it is tomorrow. I believe our senses will be heightened including auditory. Even today we can tweak our eyes to improve our vision beyond what is "normal", I would argue that "tomorrow" we will be able to add components to our bodies to heighten our senses and improve our listening abilities as well as our vision and extend it into ranges or frequencies that we don't believe we will currently need. Throwing out information that we don't believe we need today doesn't mean we will not want it tomorrow. Going back to the article, the grainy pictures of the moon landing were perfectly acceptable to the audience in the 60s, it fit their expectations and it fit their needs and ability to comprehend. But with the better pictures that were originally transmitted, perhaps we could have solved this whole lost "a" in the quote: One small step for [a] man.... Losing an [a] today may someday be equivalent to us compressing out a tone or note extension that we thought we could never hear another day but now actually desire to hear. My thought is that we have the capacity to plan for the future, let's be sure not to throw away fidelity for the sake of convenience but to be the safekeepers of the fidelity while also creating alternative forms that can be placed for convenience. Paul

Posted by Paul Monday on January 12, 2007 at 06:52 AM MST #

Post a Comment:
Comments are closed for this entry.



« July 2016