Summary of Skill Score Proposals
Received by September 15, 2011
PDF version of Duster Proposals Final
Here we present the detailed proposals that we received in September. We apologize that it took so much time to finalize these. The reason for this was that we had quite extensive dialogs with several of the proposers to tweak and evaluate the proposals to be sure that they were viable and accomplished what they set out to accomplish.
John Smaje has very kindly written a simulator for each of the skill-score proposals. Please see his proposal for a link to the simulator. Thank you, John!
Please indicate your preferences for these proposals on the doodle poll here:
http://www.doodle.com/b3mfebxn64ehrdyw
Please use your Stardust@home username when you indicate your preferences. Please be careful voting – once you have voted you will not be able to change your vote. Please vote by November 1.
Proposal A – greuti
Add a right clicked Power Movie to the calculation of the displayed Skill Score only when pre- calculation gives a higher Score than the current one. Aside from Power Movies clicked wrong would have still a negative effect to the Score (divisor increasing). But starting this calculation by a beginner it must already start with given figures, only because of not to start with a Score of 45 if the first and right clicked one is just a high rated one. For example it could be started with given Skill Score of 5 – calculated by PS(250) / PMV(50) = 5.Would then the first right clicked one count with 45 the Score would go up by PS(295) / PMV(51) = 5.78; if the next one count with 8 = 5.82; the third one with 12 = 5.94 – and so on.The displayed Power Score could go up as usual by each right clicked PM.
Proposal B – Mike Capraro
Our primary objective in revising the skill score calculation is to remove the disincentive for dusters to click on the lower-rated power movies, and eliminate the angst and frustration that some dusters have experienced seeing their skill scores drop after correctly flagging those lower-rated movies.
This proposal revises the skill score calculation so that every time a duster correctly flags a power movie, regardless of the power value, the skill score benefits. Higher-rated movies positively impact the score more than lower-rated movies, but there is never a “penalty” for correctly identifying a power movie.
The proposed approach works like this:
When a duster correctly flags a power movie, the power score increments as normal. Correctly flagging a power movie with higher value than the duster’s current skill score causes the divisor to increment by one (1.0) as normal.
For correctly flagged, lower-rated power movies, instead of the divisor incrementing by one, which negatively impacts the skill score, the divisor will increment as follows: D(n+1) = D(n) + PV/A where
D(n+1) is the new divisor
D(n) is the previous divisor
PV is the power value of the correctly flagged movie
A = 75, the maximum power movie value
Missing a power movie, of course, causes the divisor to increment by one as normal. Following are a few sample calculations for a duster starting from scratch:
|
Power Movie Value |
Divisor | Skill Score |
Power Score |
| 30 |
1 |
30 |
30 |
| 8 | 1.10666 | 34.3 | 38 |
| 22 |
1.4 |
42.9 |
60 |
| 30 |
1.8 |
50 |
90 |
| 15 |
2.0 |
52.5 | 105 |
| 5 | 2.0666 | 53.2 | 110 |
| 55 | 3.0666 | 53.8 | 165 |
| 10 |
3.2 |
54.7 | 175 |
Proposal C – Phil Greaves
To be a true indication of Skill the score should go up for each correct identification and only go down when a track/particle is missed. I believe that the Skill score should be calculated as follows:
Skill Score = Power score/cumulative missed score
My calculation has a number of advantages to the project:
- It is statistically valid – the higher the score the greater the skill of the duster in detecting tracks/particles. Being statistically valid it could not be questioned in any scientific writings.
- It increases in value with every correctly identified track (irrespective of value).
- It is simple to calculate.
- It provides a good level of feedback to the duster of ability to recognize tracks/particles and
- Gives an indication of the duster’s ability in relation to the other dusters.
To show My idea for the skill score calculation please see the following table. It shows the present way to determine the Skill Score and my revised calculation. This is a fictional dusters results for the first 45 PM’s viewed
|
PM Number |
PM Value |
Power Score |
Missed Score |
Total Missed |
Present Skill |
New Skill |
|
1 |
3 |
3 |
3.00 |
|||
|
2 |
3 |
6 |
3.00 |
|||
|
3 |
4 |
10 |
3.33 |
|||
|
4 |
9 |
19 |
4.75 |
|||
|
5 |
15 |
34 |
6.80 |
|||
|
6 |
19 |
53 |
8.83 |
|||
|
7 |
28 |
81 |
11.57 | |||
|
8 |
81 |
55 |
55 |
10.13 | 1.47 | |
|
9 |
43 |
124 |
55 |
13.78 | 2.25 | |
|
10 |
124 |
6 |
61 |
12.40 | 2.03 | |
|
11 |
3 |
127 |
61 |
11.55 | 2.08 | |
|
12 |
3 |
130 |
61 |
10.83 | 2.13 | |
|
13 |
3 |
133 |
61 |
10.23 | 2.18 | |
|
14 |
5 |
138 |
61 |
9.86 |
2.26 | |
|
15 |
138 |
8 |
69 |
9.20 |
2.00 | |
|
16 |
4 |
142 |
69 |
8.88 |
2.06 | |
|
17 |
6 |
148 |
69 |
8.71 |
2.14 | |
|
18 |
19 |
167 |
69 |
9.28 |
2.42 | |
|
19 |
12 |
179 |
69 |
9.42 |
2.59 | |
|
20 |
23 |
202 |
69 |
10.10 | 2.93 | |
|
21 |
202 |
65 |
134 |
9.62 |
1.51 | |
|
22 |
19 |
221 |
134 |
10.05 | 1.65 | |
|
23 |
3 |
224 |
134 |
9.74 |
1.67 | |
|
24 |
4 |
228 |
134 |
9.50 |
1.70 | |
|
25 |
9 |
237 |
134 |
9.48 |
1.77 | |
|
26 |
35 |
272 |
134 |
10.46 | 2.03 | |
|
27 |
272 |
19 |
153 |
10.07 | 1.78 | |
|
28 |
3 |
275 |
153 |
9.82 |
1.80 | |
|
29 |
4 |
279 |
153 |
9.62 |
1.82 | |
|
30 |
5 |
284 |
153 |
9.47 |
1.86 | |
|
31 |
5 |
289 |
153 |
9.32 |
1.89 | |
|
32 |
289 |
3 |
156 |
9.03 |
1.85 | |
|
33 |
2 |
291 |
156 |
8.82 |
1.87 | |
|
34 |
3 |
294 |
156 |
8.65 |
1.88 | |
|
35 |
6 |
300 |
156 |
8.57 |
1.92 | |
|
36 |
23 |
323 |
156 |
8.97 |
2.07 | |
|
37 |
55 |
378 |
156 |
10.22 | 2.42 | |
|
38 |
23 |
401 |
156 |
10.55 | 2.57 | |
|
39 |
27 |
428 |
156 |
10.97 | 2.74 | |
|
40 |
5 |
433 |
156 |
10.83 | 2.78 | |
|
41 |
5 |
438 |
156 |
10.68 | 2.81 | |
|
42 |
4 |
442 |
156 |
10.52 | 2.83 | |
|
43 |
4 |
446 |
156 |
10.37 | 2.86 | |
|
44 |
4 |
450 |
156 |
10.23 | 2.88 | |
|
45 |
4 |
454 |
156 |
10.09 | 2.91 |
Proposal D – greuti
One of my proposals on Skill Score would be: Every Power Movie that appears in the Virtual Microscope have to be replied in a correct way before you get another/a different one during your current or further sessions. If you do not reply correctly only this particular Power Movie will now show up again and again until one of the requested way has been clicked, i.e. you have
to make a given decision on every Power Movie to get the opportunity for more points to your scores. In this way it would be ensured for everyone that each Duster has gone the correct and traceable way by getting his scores, and did not skip low rated Power Movies. So we could leave the current Skill Score calculation as it is.
Proposal E -Joseph Moschetti
I would like to suggest the implementation of some form of feedback regarding the number of “Power Movies Missed.” I would like to know if my number of missed movies is high, low or average. I’m sure this can be done (by anonymous ranking or a percentage rating) while still maintaining each dusters’ privacy.
Proposal F – Martin St-Pierre
There are two things that I would like to suggest and I am hoping that it is technically possible to implement them (pretty sure #1 can be done but uncertain about #2):
- Give dusters the option to hide their username on the skill/power score ranking page. I think that their score should still be displayed in the Rankings page but with their username hidden.
- Split the power movies database into several tiers based on difficulty.
We all know that each power movies that we answer correctly will give us points based on how difficult it is to find the track.The amount of points that a duster will receive ranges from ~ 1 to 70.
I think that it would be a good idea to split all power movies into several tiers based on their difficulty. For example:
- Tier 1: 1 to 15 points
- Tier 2: 16 to 30 points
- Tier 3: 30 to 45 points
- Tier 4: 45 points and up
Now let’s assign a skill score range to each tiers:
Tier 1 power movies will be shown to dusters with current skill score of 0 to 5 Tier 2 power movies will be shown to dusters with current skill score of 5.01 to 9 Tier 3 power movies will be shown to dusters with current skill score of 9.01 to 14 Tier 4 power movies will be shown to dusters with current skill score of 14.01 and up I think that you can see where I am going with this now.
- This will (I think) eliminate the issue where dusters with high skill score get their score lowered after correctly answering a low point power movie (no more skipping movies).
- New dusters will be able to gradually increase their skill score without being presented with the harder power movies. Hopefully this will increase the chances of new dusters to stay active on the project without the frustration of missing nearly invisible power movies right when they start dusting.
- The points and skill score range that I have assigned to each tiers are only examples and will most likely have to be tweaked a bit to make this whole thing efficient.
Proposal G – John Smaje
My proposals are:
- Retain the present ‘Power score’, calculated as the total difficulty values of test tracks correctly clicked, as a combined indicator of skill and time invested.
- ‘Skill rating’, however, should be independent of the number of movies viewed, and not fall on correctly clicking a track of lower than average difficulty.
- I suggest skill rating should fall on missing a test track, but in inverse proportion to its difficulty. Hence missing a hard track would carry less of a penalty than missing an easy one.Given that the track difficulty values range from 2 to 75, the simplest penalty for missing one would be what I call its ‘reverse-value’, determined as (77 minus its value). Hence missing a difficult 75-value track would cost only 2 points, while missing an easy 2 -value track would cost 75 points.Overall skill rating would then be calculated as the ratio of correctly clicked values to all those viewed, where this would be the sum of the values correct plus the ‘reverse values’ of those missed.
- My formula would therefore be:Skill rating = total values correct / (total values correct + total reverse- values missed) expressed as a percentage.Mathematically, skill rating will then always rise for a correctly-clicked track in relation to its difficulty, but fall in what I believe to be a fair and understandable way if missed.
- The scientific value of such a rating system would be primarily psychological, being more likely to encourage rather than discourage participation, as well as being a spur for dusters to improve their abilities, to both their personal satisfaction and the team’s long term benefit.
NB from John: A simulation program I’ve written to test the above formula is available here (http://www.jsmaje.co.uk/SD/sd_skillscore.htm) where several of the other so far suggested formulae can be compared.
There are therefore now 8 formulae covered by the simulation:
- (= old sensitivity): skill score = number correct / number viewed
- (= current method): skill score = power score* / number viewed(* i.e. total values correct)
- skill score = power score / number viewed, excluding low values (e.g. < 19)
- (proposal B, greuti): skill score = power score / number viewed, only if rises when correct
- (proposal C1, caprarom): skill score = power score / adjusted number viewed*(* i.e. divisor increases by 1 for each correct movie only if the result rises, otherwise by its value / 75; divisor increases by 1 for every missed movie)
- (proposal C2, laserphil): skill score = power score I total values missed (x I 0)
- skill score = power score I total values viewed* (* i.e. correct + missed)
- (jsmaje): skill score = power score I (values correct+ ‘reverse values’ missed*)(* i.e. 77 minus value, thus ranging from 75 to 2, resulting in a lower penalty for missing difficult movies)