Skill Score

Discuss your experiences with and ideas about Stardust@home here.

Moderators: Stardust@home Team, DustMods

DanZ
Site Admin
Posts: 776
Joined: Fri Feb 27, 2009 3:44 pm
Location: Berkeley, CA
Contact:

Re: Skill Score

Post by DanZ » Tue Aug 16, 2011 2:54 pm

Please stay tuned for a major announcement about the Skill Score, and if you can, hold your posts until then.

Thank you!

Dan
on behalf of the entire Stardust@home team

DanZ
Site Admin
Posts: 776
Joined: Fri Feb 27, 2009 3:44 pm
Location: Berkeley, CA
Contact:

Skill Score - Request for Proposals

Post by DanZ » Thu Aug 18, 2011 6:46 pm

Dear Stardust@home Volunteers (aka Dusters)

Many of you asked for improved feedback during Phase III. In Phase IV we responded. As many of you know, we got off to a rocky start, but have now worked out the major bugs (although we admit there are some lingering issues). We thank you sincerely for your patience and for your polite communication to us regarding problems that you have found. This has been absolutely critical to fixing the problems, and we are very grateful to you for your help.

There has been quite a lot of discussion on the Forum about the new features that we implemented in Phase IV. Our guiding principle in developing all of these new features was to improve feedback so that people could have effective tools for sharpening their skills. There are varying degrees of consensus
about which are the most useful tools, and how these could be improved. In particular, we are very open to improving the new Skill Score, but we are not sure what method would be best. So we have decided to invite your proposals and put them to a Duster vote. Here is the process we are setting up:

First Step: Please send your detailed proposal to stardustproposals@ssl.berkeley.edu. We will acknowledge receipt of proposals within a day or two. Proposed changes can include: eliminate the skill score; make the skill score optional; change the skill score formula; and so on. You are welcome to suggest multiple changes, of course, and we are open to any ideas that could improve the project. [Note: If you are not comfortable writing in English, please send the proposal in the language that you are most comfortable writing in, and we will translate.]

Step Two: We will do a preliminary evaluation for feasibility and science impact. This evaluation will focus only on these two aspects and we will not make any judgments beyond these. If your proposal is not feasible, or if we are concerned about a negative impact on the science, we will send it back to you with detailed comments, and you will have an opportunity to revise and resubmit.

Step Three: On September 15, 2011, we will close the process to any new proposals, and will post the feasible proposals received on the Stardust@home website.

Step Four: We will ask you to vote on proposals. In September we will set this up. It will probably be a poll on the Stardust@home website. We will close the voting process on October 15th. If there is not a clear winner, we may have to have a runoff.

Final Step: We will implement the consensus recommendation as soon as possible. If it is a simple change it may happen quickly, but if it is more complex, it make take more time.

Requirements for your proposals:

You should be very specific in your proposal, and should explain the reasons for the changes that you propose. It should always come back to this: How will your proposed change improve the Stardust@home project? Remember that the other dusters will be reading and evaluating your proposal, so you will need to be very clear. Please use simple/concise statements (especially since not all of the dusters are English speakers).

We are looking forward to your ideas!

The Stardust@home team

laserphil
Posts: 81
Joined: Sat Dec 02, 2006 9:28 pm
Location: Sydney Australia

Re: Skill Score

Post by laserphil » Wed Oct 05, 2011 4:04 am

Is there any news about the skill score submissions?

DanZ
Site Admin
Posts: 776
Joined: Fri Feb 27, 2009 3:44 pm
Location: Berkeley, CA
Contact:

Re: Skill Score

Post by DanZ » Wed Oct 05, 2011 6:40 pm

laserphil wrote:Is there any news about the skill score submissions?
Yes, it's getting close and Dr. Westphal was "negotiating" with some of the proposers on alternative ideas. I will send a request for an update. Thanks for your patience!

Dan

fjgiie
DustMod
Posts: 1253
Joined: Sat May 20, 2006 8:47 am
Location: Hampton, SC, US

Re: Skill Score

Post by fjgiie » Thu Oct 20, 2011 6:20 am

@ greuti

It appears that fjgiie did not understand what greuti was talking about when he said "right click". Right click meant to correctly click on a Power Movie, not right click the mouse. Am I correct about this? We don't need to right click the mouse, correct?

           fjgiie

greuti
Posts: 200
Joined: Fri Aug 11, 2006 10:11 am
Location: Switzerland

Re: Skill Score

Post by greuti » Fri Oct 21, 2011 1:06 am

Hi fjgiie

Yes, I've meant a correct click on a Power Movie.

Now your sentence makes more sense to me too :)
We don't want to right click anything as the Stardust computer can take care of this when updating the Skill Score after every clicked movie.

laserphil
Posts: 81
Joined: Sat Dec 02, 2006 9:28 pm
Location: Sydney Australia

Re: Skill Score

Post by laserphil » Mon Oct 24, 2011 4:01 am

With all the complaints about the Skill Score I would have expected that more than 11 dusters would have voted by now.
I was one of those who complained, I submitted a proposal, I have cast my vote.
It looks like those who demanded change have already departed and it has been left to the ones who put in submissions to get the change happening.
Come on dusters - make your vote count you only have until November 1.
If you really want the Skill Score to change the Stardust team will want a lot more than 11 votes.

Laserphil

elainekeefe
Posts: 190
Joined: Fri Aug 04, 2006 4:38 am
Location: Massachusetts, USA

Re: Skill Score

Post by elainekeefe » Mon Oct 24, 2011 6:02 am

laserphil wrote:With all the complaints about the Skill Score I would have expected that more than 11 dusters would have voted by now.
I was one of those who complained, I submitted a proposal, I have cast my vote.
It looks like those who demanded change have already departed and it has been left to the ones who put in submissions to get the change happening.
Come on dusters - make your vote count you only have until November 1.
If you really want the Skill Score to change the Stardust team will want a lot more than 11 votes.

Laserphil
There's voting? I would have expected it to be posted here as well as on the Stardust site. I imagine many people didn't even realize it had commenced. I know I didn't. Maybe now that people know, they will vote.

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill Score

Post by jsmaje » Tue Oct 25, 2011 12:11 pm

Agreed, laserphil & Elaine!
Given that the team officially designated this forum topic to Skill Score issues, at least a link to the News/Blog item and voting opportunity should have been posted here. I find it all too easy to skip the Home page, given the rarity of anything new of interest, instead going straight to the forum for the occasional gem.

I too hope that there will be more than 11 votes, but since there are now only 214 world-wide participants currently on the power score list, and 233 on the skill list (I think the former number is lower because power score needs to be > 10 before inclusion?), and only around half of the Top 100 scorers have been active over recent weeks, I reckon a final tally of 20 to 30 duster votes to be optimistic.

Meanwhile can I suggest they make use of my simulation of skill-score formulae, as endorsed by the team, before making their decision(s)!
John

laserphil
Posts: 81
Joined: Sat Dec 02, 2006 9:28 pm
Location: Sydney Australia

Re: Skill Score

Post by laserphil » Wed Oct 26, 2011 3:42 am

Hi John,

I like the way that you can develop your computer simulations and the graphical representations of the top 100.

Can you do a simulation of the Skill Score presentations to show how they remove the backwards movement of the skill score when a track of lower value is correctly identified - I believe that this was the main problem with the present calculation and I cannot see it from your current simulations?
The last two nights of dusting have shown just how painful this can be for me. I started at 14.45 and have managed to work my way down to 14.40 via 14.33. I did miss a couple of tracks but I don't think I deserved the five hundred points in 2 to 15 point tracks it took to get back.

Also wondering about the (XI0) after my proposal?

Best regards
Phil

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill Score

Post by jsmaje » Wed Oct 26, 2011 2:15 pm

Thanks for your interest Phil.

Yes, I’m sorry that my Skill-score simulation demos don’t really demonstrate the problem with low-valued movies all that obviously (though it may depend on your screen resolution*). Close examination will show a slow fall in skill level during the low-value delivery segment when using the current formula (2) as opposed to flat-lining with fjgiie’s (3) for example, and various other behaviours with the others.

One way to highlight the issue is to set duster accuracy to 100% throughout the whole range of difficulties using the Custom settings (it’s worth experimenting with all of these, that’s why I optimistically included them!). Despite now being correct all the time, and if using the present formula (2), on being presented with lower-valued movies than one’s average skill level, the latter inevitably falls, as opposed for example to fjgiie’s (3) which simply ignores such movies, as these graph extracts below show:

Image

Regarding that times-10 factor, it was simply because your formula tends to produce rather small figures, as your own data demonstrate, while others result in higher figures or percentages that could be conveniently fitted to the common 0-100 graph y-axis I settled on for the sim. Otherwise, your fractionally-subtle results would burble inconspicuously close to the bottom of the graph. I just wanted to make them more obvious on the chosen graph scale, which Andrew Westphal agreed was sensible.

Hope that helps,
John
* site made for my own 1280 x 1024 resolution; apologies to everyone else if that causes problems.

laserphil
Posts: 81
Joined: Sat Dec 02, 2006 9:28 pm
Location: Sydney Australia

Re: Skill Score

Post by laserphil » Fri Oct 28, 2011 10:24 pm

Thanks for the explanations John, I could see the effect on the simulations of the low scored PM's but was hoping that it would stand out a little more.

I don't know for sure but my gut feeling is that there are considerably more low value PM's than high value in use so your simulations may be skewed towards higher skill values. I was surprised to see that my proposal gave such low scores that a 10X factor was required. The calculation should return any value between 0 and infinity depending on the dusters "Skill".

Unfortunately my proposal yields an undefined result for 100% accuracy as the denominator is zero. Maybe some dusters would like an infinite Skill score! You must have converted my results to a percentage for the simulation with 100% accuracy!

All the best
Phil

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill Score

Post by jsmaje » Sat Oct 29, 2011 1:50 pm

OK Phil, have I interpreted your formula correctly?
Skill Score = Power score/cumulative missed score (where Power score = cumulative correct score).
Hence, Skill Score = cumulative correct score/ cumulative missed score.

I have to admit that my pre-set demos don’t reflect the likelihood of increasing duster accuracy over time, which I guess your formula may well do, but this option is in fact provided via the Custom Settings.
And regarding the effects of lower-than-average difficulty movies, look at the wildly-oscillating skill score during the first 200 demo deliveries before settling to a less variable average for most formulae.

I wish I’d allowed myself more time before the proposal deadline to finesse the sim and pass it by such as yourself, so as to provide more informative demos, including long-term comparisons of each formula regarding increase in duster accuracy for example.
Whatever, there are only a few days left before closure of voting and, sadly, it seems very few remaining dusters all that interested anyway!

I also confess that each time I’ve naively delved into this project’s stats and other issues such as this over the last 5+ years, the more complicated they’ve turned out to be, and my sympathy must ultimately be with the team, whose priority has of course to be the basic interstellar particle science - of which It would be good to hear more!

John

PS: can't say I've noticed lower-valued test movies being more frequent than higher, though they're more obvious of course.
Can Dan /Andrew / tech team clarify?

laserphil
Posts: 81
Joined: Sat Dec 02, 2006 9:28 pm
Location: Sydney Australia

Re: Skill Score

Post by laserphil » Thu Nov 03, 2011 4:02 am

Thanks for your explanations John. Yes you have the formula correct. I did not include the statements that a duster would have to have at least, found one track and missed one before my formula would give sensible values otherwise zero and infinity would be the results.

Just as an aside I completed a session of 220 movies tonight so I would expect to have had around 40 PM's. In that 40 odd PM's I had 2 of value >50 - I know, because I missed both. I started the night with a skill score of 14.50 lost .04 for the two high score movies then proceeded to work my way down to 14.41 before encountering two PM's of value 40 and 16 which brought me back up to 14.45.
During the whole session I received only 3 PM's with a high value, the other 37 were either < or close to my skill score.

This is what I was referring to with the comment about the incidence of low value PM's. It may just be the luck of the draw for me but it is really frustrating as it occurs over and over like this!

Phil

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill Score

Post by jsmaje » Sat Nov 05, 2011 2:23 pm

laserphil wrote:Thanks for your explanations John. Yes you have the formula correct. I did not include the statements that a duster would have to have at least, found one track and missed one before my formula would give sensible values otherwise zero and infinity would be the results.
Yes Phil, I was aware of that, and my simulation of your formula takes this into account. The actual javascript code for your formula in the sim is as follows, which I hope to be self-explanatory:

if (valMissed == 0) skillScore = 100; // i.e. avoid infinity
else if ((skillScore = (valCorrect / valMissed) * 10) > 100) skillScore = 100; // i.e. calculate, but cap at screen graph maximum

Ten test runs of your formula, starting with my default setting of 100->10% duster accuracy over the movie difficulty range, but improving to 100->100% over 10 runs, have all ended up with figures from 22.85 to 23.92.

However, since the sim does in fact assume a random distribution of movie-difficulties (from 2-75) over time as I believed, its results may be biased were this assumption to be incorrect.

I’m therefore keen to hear from the team regarding my last post’s PS regarding this difficulty-range distribution issue that your formula (as you point out) would clearly be sensitive to.
I did submit the sim to Andrew Westphal though, asking that his techs first check it out before providing the link!

John

Post Reply