To find cheaters, we compared the median evaluation time spent on real movies to the median evaluation time spent on calibration movies for the top 200 ranked searchers. Cheaters would have a very much smaller evaluation time for real movies (because they're just flipping through them quickly) than for calibration movies. This makes them easy for us to spot.
To demonstrate, we show two sets of histograms below -- one is for a cheater, the other for a true collaborator.
In each pair of histograms, the top figure shows the distribution of evaluation times for calibration movies, and the bottom shows the distribution of evaluation times for real data.
A real collaborator will treat calibration and real movies almost the same way -- if anything, the times will be shorter on average for the calibration data, because some tracks are obvious and can be discovered quickly. A cheater, on the other hand, will have much shorter evaluation times for the real data. Can you tell who is who?
As we had hoped, we found that out of the top 200 volunteers in the rankings, only eight were cheating. Everyone else was simply incredibly hard-working and talented!
For the cheaters, we are ignoring their data and have zeroed their scores. If they would like to do some real scanning, we invite them back, and suggest that they re-register under a different username.
1
2
Stardust@home team