Skill score decreasing for found movies

Post here if you are having any kind of problem with the Stardust@home website.

Moderators: Stardust@home Team, DustMods

jasonjason
Posts: 44
Joined: Mon Jan 28, 2013 6:34 pm

Skill score decreasing for found movies

Post by jasonjason » Sun Feb 09, 2014 1:46 am

Does anyone have an answer to why my skill score decreses when I FIND power movies? In some instances I find three in a row and each time it drops my score and drops my rank, this is real frustrating, it is like getting penalized for finding movies. The better you get the worst your score gets. It works fine up till you find the 100 power movies thereafter it seems not to work.

Help on this frustrating matter will be appreciated.

Jason

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Sun Feb 09, 2014 1:16 pm

jason/jason, I relate fully to your problem. The reasons for Skill Score descending in response to finding tracks is fully explained under "jitter simulation" topics. Check under "Stardust Problems & Support," --"Answers to questions about movies, scores, etc." and read my post on 11/26/13 and 12/27/13. My misery might find good company with you. The "jitter simulation" posts are in depth with charts and graphs, that you will find interesting. But, the fact that it can be explained does not make it right. If one finds a track, then credit should be given, not taken away. As I said in another post, I am an old lab-tech gal. Worked for WVU and OSU. My skills were regularly checked by ASTM samples ---- and my job skills were based on how many ASTM samples I got right. (Fortunately, I never failed an ASTM test) It is inconceivable to me that my job performance rating would have gone down when I correctly analysed a sample! Yet, that is the norm in our Phase 6. Makes me sick every time it happens -- and it happens daily! My condolences to you, my fellow duster! Evelyn (ERSTRS, SmithES)

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Sun Feb 09, 2014 1:43 pm

Jason, My dusting experience today was: Began with 9168 Skill Score and 17385 Power Score. Found and identified 4 tracks. (No errors) Ended up with Skill Score of .9157, a drop of .0011 points, while Power Score rose to 17500---a rise of 115 points. No chart, jitter or otherwise, can make that right. But, there's nothing you or I can do about it. I've been dusting since August 2006, and no other Phase used that formula to derive Skill Score. If I hadn't been around so long and become an entrenched "family member," I would quit. Evelyn (ERSTRS, SmithES)

jasonjason
Posts: 44
Joined: Mon Jan 28, 2013 6:34 pm

Re: Skill score decreasing for found movies

Post by jasonjason » Sun Feb 09, 2014 5:10 pm

The question is is it going to be rectified, can't they go back to scoring the old way :?:

Thanks for the quick reply

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Mon Feb 10, 2014 7:04 am

Jason: The question is is it going to be rectified, can't they go back to scoring the old way :?:

Jason, To answer your question, I SURELY HOPE SO! Stardust@home is unique in many ways. However, I know of no other business, organization, or system that penalizes folks for doing something right. Evelyn

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Thu Feb 13, 2014 12:15 pm

Hi, Jason, The problem continues. Three days ago my Skill Score was .9170, giving me hope I could move upward from #15 listing. Today, 3 days later, I just did some dusting, and after finding four tracks and making no errors, my Skill Score dropped to .9154. I don't want to continue dusting today or I'll find myself in the cellar. Will try to claw my way back up tomorrow. Evelyn (SmithES, ERSTRS)

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Fri Feb 14, 2014 6:22 am

Hey, Jason, How YOU doin' on YOUR Skill Score? Looks like you and I are the only two complaining. I checked in this morning with S.S. of .9154. Found three tracks, no errors, and by golly---my S.S. is now .9122. Penalized again for finding three tracks, I quit for the day. :( Evelyn (ERSTRS)

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill score decreasing for found movies

Post by jsmaje » Fri Feb 14, 2014 1:58 pm

Hi Evelyn (& Jason). Believe me, I share your annoyance with drops in skill score on correctly identifying a PM track when this happens to coincide with a higher-valued PM dropping out of the moving average window. But you’ll recall that dusters were already expressing frustration with the ‘specificity’ (or was it ‘sensitivity’) and later ‘skill score’ methods at least back to 2011, including in fact yourself: “When will the scoring system be changed?” !

The reason was that skill was then calculated as a cumulative average from the beginning of the phase, which would not only (even then) result in a fall in score when getting right a particularly easy track compared to the previous average, but significantly punish missing a difficult track (hence the practice of ‘skipping’ such movies that developed), as well as being very slow to reflect any increase in personal skill. Should you have made many mistakes early on, there would be a very long climb up the ranks.

Dan encapsulated the issue here, his message in fact immediately preceding your own above!

Hence the search for both a fairer skill formula and means of implementation. While my own formula is presently in use, the relevant issue here is that a moving window of 100 PMs was also applied as a means to address the demand for a quicker response to increasing personal skill. Just like any mathematical average, including cumulative, a moving average inevitably incurs the property of what may be considered unexpected drops, but also an equal number of rises to the same degree overall. Any skill score formulation would be subject to the same mathematical phenomenon.

To quote your post: “my job skills were based on how many ASTM samples I got right. (Fortunately, I never failed an ASTM test). It is inconceivable to me that my job performance rating would have gone down when I correctly analysed a sample!” That suggests a simple tally was made of correct results, with no account taken of relative difficulty, and (since you admirably got everything right) no way of assessing increasing skill, both being features asked for by dusters in previous phases.
Are you really sure you want such a simple tally method, or a return to the previous cumulative average which would continue to produce occasional unwelcome drops (albeit of lesser degree) but at the same time be very sluggish and incompletely respond to changes in skill level?

My suggestion of what may be called a ‘lead-in period’ by down-weighting the oldest PM scores in a moving window is an attempt at a best compromise given the inevitable mathematical reality, long-known by statisticians and as used in many other professional applications.

John

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Sat Feb 15, 2014 5:37 pm

John (jsmaje) I am very appreciative of your consideration for my concerns, and since only two of us are complaining, I think I should just drop it, and roll with the punches. It's the bottom line that bothers me--- punished for getting something right. And then, today---GLORY BE---I missed a 70 pointer and my Skill Score went UP! First time that's happened to me. But it shows the reason for my puzzlement: I missed a big track, yet I was rewarded for missing it! Something will always bother me about a system that produces those results. As for the labs at WVU and OSU, the rating systems were very simple as you indicated. Percentages were based on how many American Society for Testing and Materials samples one got right out of the samples received. Of the usual dozen lab-techs I worked with, 100% right was not unusual. After all, if one did a thorough job of testing the gas, water and coal samples, there were few mistakes. The municipal Water, Coal, and gas companies, depended on us for correct answers. (Lives could be at stake, in fact) A monthly sheet of scores was displayed, listing each person's skill, determined by how many they got right compared to how many wrong answers. The ASTM samples served another purpose--- if we got an incorrect result on one of their samples, most of the time it was because some of our equipment needed re-calibrating---not because of an error by a lab-tech. I also was a Lab-tech for Union Carbide and Carbon, and they used the same ratio right/wrong test results, to determine one's skill. I'll never get used to being punished for a right answer, and then rewarded for a wrong answer! That last one blew me away!

Therefore, please just leave the present rating system alone. I may groan about it, but I'll quit growling about it. Evelyn (ERSTRS, SmithES, August 2006)

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill score decreasing for found movies

Post by jsmaje » Sun Feb 16, 2014 1:39 pm

Well, there you go Elaine! You make the point for me, that unexpected increases in skill score will occur despite having missed a PM track which happens to coincide with a previously-missed PM of lower value dropping out of the window. This will actually happen as often over time as unwelcome drops, and to the same overall degree, so it all balances out in the end. It’s just that you, indeed all of us, will be more likely to notice and mentally register the annoying drops than rises.

So, although this might have been the first time you’ve become aware of this, you can be sure that it’s been happening throughout your time dusting. The ‘jitter’ sim provides a run of 1000 PMs, which in ‘Holding’ mode (+- Pause) shows the details of such unexpected events, that in the sim (as written) occur about 12% of the time (6% unexpected falls, 6% rises). You may also try keeping a record of your own correct/incorrect decisions vs skill score change for a while to convince yourself. There is really no reason to ever consider yourself being "punished" by a statistical reality, to which we are all subject.

And use of a moving average window has its advantages that to my mind outweigh its disadvantages, though others may disagree …

John

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Sun Feb 16, 2014 5:17 pm

Hey, jsmaje, Let me respectfully remind you that my name is EVELYN, not Elaine. . . . Also, I keep meticulous records. How else do you think I can report exact numbers in my posts? That WAS the first time my skill score has advanced with a miss. . . . EVELYN

jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Skill score decreasing for found movies

Post by jsmaje » Mon Feb 17, 2014 2:40 pm

Oh dear, sorry to have offended you Evelyn. You may remember I’ve confused your name with Elaine (Keefe) at least once before; I’ve always been hopeless with names, and as I get older it only gets worse.

Further in my defence, your present excellent power score of 17690, if divided by an average PM value of say 40, implies about 445 PMs to have been viewed, a bit less than half the (admittedly long for statistical reasons) 1000 sim movie run. Nonetheless the sim would have predicted about 6% = 26 unexpected score rises by now, so it could well be that my program’s assumptions and &/or implementation are unrealistic, for reasons readily acknowledged in my postings.

However, all statistical math inevitably predicts symmetric ‘jitter’ over sufficient time in the absence of biasing/disruptive effects. How in fact could it be otherwise?
Your own experience may be due to one of those extreme statistical variations which are bound to occasionally occur (also Jason, though from whom we’ve heard no more so far).
Can we know your recorded number of unexpected drops vs PMs viewed to compare with the one unexpected rise so far, and for you to continue doing so, say once a week or month? And it would also be interesting to hear from others who may have kept records but not published them.
If such an asymmetry continues and is shared by others, then my understanding of the situation must be wrong, or something else in the score calculation implementation is awry.

The team have all the data to hand, of course, but they sadly always seem too busy/reluctant to share them with their volunteer collaborators in my experience.

Best regards,
John

SmithES
Posts: 186
Joined: Sun Apr 17, 2011 5:25 am

Re: Skill score decreasing for found movies

Post by SmithES » Mon Feb 17, 2014 5:27 pm

Hi, John, I am in a better mood right now. I apologize---this old lady can get testy late in the evenings. (Confusing me with Elaine Keefe is a compliment---I shouldn't complain---she was one fine duster, and I miss her.)
John, I've viewed 749 PMs, in fact. Yes, I will send you my records once each week. I'll try to figure out an easy way of recording them, in a chart or something. Evelyn

February 17, 2014
Power Score: 17690
Current Skill: 0.9139
Movies Viewed: 4569
Power Movies: 749
Movies Missed: 50

caprarom
Posts: 336
Joined: Thu Aug 02, 2007 7:12 am
Location: Riverview, MI

Re: Skill score decreasing for found movies

Post by caprarom » Mon Feb 17, 2014 5:59 pm

Rats! Double submittal. How do I delete this message? Oh well, just see the next (previous?) one.
Last edited by caprarom on Mon Feb 17, 2014 6:06 pm, edited 1 time in total.

caprarom
Posts: 336
Joined: Thu Aug 02, 2007 7:12 am
Location: Riverview, MI

Re: Skill score decreasing for found movies

Post by caprarom » Mon Feb 17, 2014 6:03 pm

Hey, Evelyn, I've also missed (or flubbed) 50 PMs in Phase VI so far - nice round number. A pleasure to be in your company. My missed count might go up soon as I've been traveling the last four days and am now a bit out of practice. Always nice to have an excuse, anyway. Happy dusting - don't let the 'jitters' get you down.

Mike C.

Post Reply