Thursday May 20, 2010
Tuesday Sep 22, 2009
By eoin on Sep 22, 2009
Netflix Prize awardedThe Netflix Prize was finally awarded three years after the competition began. A team of seven individuals (BellKor's Pragmatic Chaos) scooped the $1 million prize.
Netflix is a movie rental company, and in 2006 were looking for a way to improve their movie recommendation system. This was important to them, as it is to Amazon and pretty much every other online retailer, since it allows customers quickly find products that will appeal to them, and hence increase sales and return custom.
They released a database of 100 million movie ratings from almost half a million customers. The challenge was to predict the ratings these users would give to a quiz dataset of movies. Netflix knew and withheld these ratings and marked competitors by how closely their predictions match the real scores that customers gave to withheld ratings.
The competition is important in that it significantly advanced the state of the art in recommender systems (aka collaborative filtering) in an open and collaborative manner.
A whole host of techniques were either developed or enhanced to tackle the problem and to handle the large size of the database - the largest publically available database of customer rating data.
One interesting outcome was that no single method of analysing the data was sufficient to achieve the required quality of predictions. Combinations or ensembles of many different techniques provided the best results.
All the details, including the papers describing the winning algorithms, are available at the Netflix Prize website.
Dead Fish responds to Human EmotionThis story has been doing the rounds of Slashdot and the Register recently, but is pertinent here given my interest in statistics.
The researchers in human brain mapping used the standard statistical tools for analysing MRI data of human brains and applied it to a fish. A dead Atlantic salmon to be exact. The test otherwise followed normal lines: "the salmon was shown a series of photographs depicting human individuals in social situations. The salmon was asked to determine what emotion the individual in the photo must have been experiencing."
Afterwards the found correlations between brain activity as measured by the MRI equipment and analysed using standard techniques, and the photographs of the people. Thus proving that a dead fish responds to the emotions displayed by humans in social situations.... or perhaps that the standard statistical analysis needs more rigour.
The original poster is online.
Snippets of code, useful tips, and observations of working with Sun.