Weekly trends

[DTF]

John,

I was thinking about an alternative method of graphing these statistics, when it hit me that this might well lend itself to a graphing similar to an IQ graph. The typical IQ bell shaped curve is the result of mapping the number of participants with a particular IQ on the verticle axis and the IQ score on the horizontal axis. I would think that doing a similar mapping of the members would result in a very skewed bell curve, which would be less skewed in phase 1 than it would be in phase 2. This could involve just the top 100 participants, if you are so inclined, or all of the participants or any kind of subset in between. I'm not at all sure that it would result in anything truly meaningful, though it might be interesting to look at.

[jsmaje]

Forgive me if I'm wrong about this, but my understanding was that the scoring system was originally implemented in order to place greater mathematical importance to the members identifying tracks in actual movies who were more acurate in identifying tracks in test movies.

Well yes, that's what the team initially said, but it seems they've subsequently decided to put particular faith in the top 23 rankers (+ 6 staff) to prioritise candidate tracks for extraction, i.e. the 'Red Team' (at least those they approached who agreed to be members). To my knowledge, no mention was made that their accuracy in terms of sensitivity or specificity stats was ever taken into account. One would hope there'd be a reasonable degree of correlation, but I wouldn't have thought it mathematically guaranteed. Meanwhile, I have to say that the stats published by the few that have are indeed very impressive (see Post your stats), but what about the rest who haven't published them (nor indeed taken much, if any, part in the forum)? And the list of RedTeam members is now apparently closed to new entrants; so for the present that's that as far anyone else having any stats-related influence.

I therefore, tend to think that the ranking of members came more from the recognition of its importance to members participating in the forum

Well hardly: scoring and ranking were built into the project from the outset, way back in ?1863

I was thinking about an alternative method of graphing these statistics

I also briefly wondered whether what we're looking at here is the top end of a bell curve set on its side, but there are problems with this. Looking at the raw data from phase 1 scores suggests that such a curve would have to be extremely left-skewed, with no significant or related (nor a priori logical?) relationship between median, mode and mean, i.e. not a Gaussian-type curve worth that name at all. Without plotting all 21,070 points I can't be sure. Nor am I going to bother, though suggest the team consider it worth setting as a student punishment excercise!

Whatever the appropriate formula, and the various constants and variables relevant to this sort of distribution, surely they must have a behaviourable basis.

John

[DTF]

Hi John,

You are unbelievably correct about the IQ graph idea. It would absolutely be a meaningless exercise. I thought that I could easily group scores into even intervals across the board for the phase 1 figures, and that is indeed possible without much labor involved. However, since there are so many individual scores below 500, or even 1000, and so comparatively few above those points, the end result would be a graph that starts off at the highest point on it and the rest of the points would be so close to zero, or would actually be zero, that you could not tell them apart.

[jsmaje]

OK, I think I have an answer to my question about the particular shape of the Stardust Top 100 score vs. ranking curve, and it's more interesting than I'd ever expected because of the fascinatingly wide range of phenomena found to exhibit the same distribution for reasons not always apparent.

Pier was absolutely right that it seems to be a variety of what is sometimes called a ‘shifted’ power-law relationship with the formula y=a(x-b)^c, though I’ve found it more often expressed in the form y=a/((b+x)^c), known as a Zipf-Mandelbrot (ZM) discrete probability distribution (the formulae are mathematically equivalent as long as constant a is positive and b & c negative in the first while all are positive in the second).

This is a generalisation of Zipf's law, initially proposed in the 1930’s to model word usage in English texts, frequency being almost inversely proportional to rank, i.e. y=a/(x^c) with exponent c ~= 1.0 (or –1.0 in the first formula), and closely related to the Pareto distribution (initially regarding wealth-distribution in Italy), aka the "80-20 rule" or “Law of the vital few” (a widely-used management and quality-control rule-of-thumb, e.g. 80% of income comes from 20% of clients, or faults from potential causes, etc.).

If truly ‘Zipfian’, a log-log plot should be entirely linear with slope ~= -1.0. Here are some recent examples regarding (A) word frequency in a variety of English texts, (B) website popularity in Russia, (C) note pitch distribution in Bach’s ‘Air on the G string’(!), and for comparison (D) the SD@H full list of Phase 1 scores. In each case the law indeed seems to apply remarkably well throughout most of the range, except for the ‘droopy’ tops and tails:



In order to generalise the formula, Mandelbrot added another (arbitrary) constant b to the divisor in Zipf’s original, (i.e. giving y=a/(b+x)^c), which assumes more significance the lower the rank number becomes, thus modelling the top-end droop. The authors of graphs A & B suggest further (but different) modifications to account for the tails also consistently evident in their data.

Although figure D shows that the top 100 SD@H scores alone fall largely within the top-end drooping region, the log-log plot of all Phase 1 scores (534 dusters) does show an overall slope of almost exactly -1.0 (using the first version of the formula in CurveExpert 1.3 as pier suggested, with a=1075444.1, b=-6.8524834, c=-0.86919775.)

What this may mean at any deeper level I’m not sure, but have always felt certain there had to be something to it, and suggest SD@H can now be added to the growing range of ZM phenomena including not only the examples above but also global manufacturing, earthquake time intervals, the magneto-rheological properties of ferrofluids (whatever that may be), and many many more...

John

[Nikita]

So in other words, we are simply doing what we do as part of the ZM mathmatical equation - Hmmm....is this the mathmatical constant for all of life???

I'm not sure what your results may indicate, and it looks like it took a lot of work to make a curve to fit the data. Thanks for your efforts!

Now we just need this curve to tell us where the dust is....

[jsmaje]

So in other words, we are simply doing what we do as part of the ZM mathmatical equation...?

Well yes, but only in a sense; it looks to be one of those 'laws of large numbers', like a Poisson distribution, or the more familiar bell (Gaussian) curve regarding people's height, IQ etc., in no way challenging free-will or whatever. If dusters reduce or increase their scores, drop out or join in (as they do), the details of the curve may change, but not it's overall shape, as long as - and I think this is the relevant point - each duster operates independently of the rest.

It wasn't difficult finding the constants and fitting the curve using CurveExpert (thanks to pier). It took me longer to explore how widely the ZM formula and its several modifications apply; in particular, I was astonished after entering the data how similar the log-log plot and range of phase 1 scores was to Shakespeare's works and website popularity! Who would have thought it?

Now we just need this curve to tell us where the dust is....

If only

[Ver Greeneyes]

Interesting phenomena.. Have you considered that the tapering off at the top of the graph maybe simply due to physical limitations? i.e. the number of hours in a day, the quality of the images, the limit of human shape recognition.. If not for those, perhaps the graph would be more-or-less perfect, and it might be interesting to figure out how significant each factor is to the tapering off. Hell, if there was enough data, maybe you could even draw some meaningful conclusions about human ability by assuming a perfect ZM relationship and comparing it to reality..

[pier]

Interesting phenomena.. Have you considered that the tapering off at the top of the graph maybe simply due to physical limitations? i.e. the number of hours in a day, the quality of the images, the limit of human shape recognition.. If not for those, perhaps the graph would be more-or-less perfect, and it might be interesting to figure out how significant each factor is to the tapering off. Hell, if there was enough data, maybe you could even draw some meaningful conclusions about human ability by assuming a perfect ZM relationship and comparing it to reality..

I'm finding surprising the fact that the behaviour of the "Top 100" is comparable with that of much wider population.
Actually 100 in this contests is a very little number.

Pier

[jsmaje]

Have you considered that the tapering off at the top of the graph maybe simply due to physical limitations? i.e. the number of hours in a day, the quality of the images, the limit of human shape recognition..

Yes, I think that's the general explanation whatever the particular phenomenon and relevant factors concerned; there'll obviously be some absolute limit to the total number of words in a language, capacity of websites, and other sorts of practical limitations such as you mention. Similar arguments apply to the tail.

What I find surprising (and to my knowledge remains unexplained) is that the prevailing log-log slope should nevertheless be so close to exactly minus 1.0 for so many disparate phenomena; why not -2, e, pi, some other arbitrary number, or indeed any consistent log-log relationship at all?

And I share pier's surprise that the slope of the Phase 2 scores for the first 100 rankers in graph E (equivalent to the phase 1 dashed box of graph D) approaches so close to -1.0 after having excluded only the top 12 dusters (-1.146 to be exact, i.e. slightly but only marginally tilted compared to -1.029 for the entire corpus of 21,070 phase 1 participants). It would be good if the team could make all the on-going phase 2 scores available to confirm the pattern, but I predict it would differ little from phase 1.

I find the Russian website-visit graph (B) particularly interesting, showing the tail-end starting significantly later according to increasing year, presumably correlating with the growing number of surfers, while the intermediate slope remains stubbornly at around -1.0.

I like the idea of extrapolating to theoretical perfection; the graphs suggest that only one sharp-eyed duster could have easily evaluated all ultimate million-ish Stardust movies by now if working at maximum rate 24 hrs a day (KAC has of course been doing his best in this regard , though at present seems to be taking a well-earned holiday at 222222).

John

[jsmaje]

Although the project (or at least the forum) has gone quiet lately and the present Top 100weekly activity graph (below) shows a continuing drop since first extraction, I found Andrew's 4th ISPE updatequite reassuring.



Not clear to me, though, is whether we've evaluated the top-left 1/3 of the collector tiles during phases I & II to a sufficient, or perhaps even more than sufficient, extent; apparently there are now over 100 candidate tracks awaiting extraction. Would it matter, for example, if all dusters were to take a break until the promised scanning of new tiles later this year?

John

[DanZ]

Not clear to me, though, is whether we've evaluated the top-left 1/3 of the collector tiles during phases I & II to a sufficient, or perhaps even more than sufficient, extent; apparently there are now over 100 candidate tracks awaiting extraction. Would it matter, for example, if all dusters were to take a break until the promised scanning of new tiles later this year? John

Officially the team prefers if people keep searching (although, note Caveat below). You see, although the images have been well-searched, everyone is different, and even one person may spot something that others have missed. No data are ignored, although to be sure it may take more time to look at something that only one person has flagged. Caveat: one should only be doing this if one is having fun -- if you're not having fun doing it, stop.

Hope that helps!

Dan

[jsmaje]

Despite the apparent present lull in scientific Stardust activity, I've noticed a recent resurgence in Top 100 dusters' activity. Following the trough of 29% in April vs. initial phase 2 activity shown in my last posting above, there has since been a clear (albeit wobbly) increase over the last 15 weeks up to a respectable 68%:



I can't relate this to holiday times or whatever, so what else has changed? May I suggest at least one factor: DanZ's appointment during this period, and his willingness to quickly respond to forum questions (something long lacking and often complained about), hence encouraging rather than discouraging participation. Long may this continue.

John

[DanZ]

Hey thanks John, I'd be honored if I'm having a positive effect. FYI, I think new scans are actually in the works and will try to get an update for you soon.

Dan

[jsmaje]

I present without comment the end-of-year (pre-Christmas-NewYear-holiday) summary of weekly activity amongst the Top 100 scorers:



Roll on phase 3 - any firmer news Dan?

Meanwhile, Happy Christmas, Yuletide, Hanukkah, or whatever way you have of marking the winter solstice.
John

[DanZ]

Thanks for that update John.

All I can say at this point is that you will definitely be hearing from us in January, that there will be a Phase 3, and things will definitely be different (although not exactly as we originally thought). More soon!

In the meantime, here's hoping everyone has a wonderful holiday season!

Dan