Scoring method

Discuss your experiences with and ideas about Stardust@home here.

Moderators: Stardust@home Team, DustMods

Sasari
Posts: 44
Joined: Mon Jul 31, 2006 3:17 pm

Scoring method

Post by Sasari »

The current scoring method is blatently open for major abuse. If you see ondeck_movie_type=testing in the link you know it is a test video. You can spend your time looking for a track. If it says real you are safe to just click no track.

Easily gaining a lot of points. I'm sure you could get even more creative with how the links work to give yourself testing movies on purpose. I suggest a system that doesn't let the user directly know if its a test video or real data.
Whity
Posts: 23
Joined: Wed Jun 28, 2006 4:29 am

Post by Whity »

true,

if you look at some participants rate of scoring you get suspicious..
imho some people already have made scripts that contain the answer to the testing movies and don't have to wait for the images to load before submitting the answer.

Not that I can think of any reason that people want to cheat such a project but weird people do exist.

Maybe some kind of trapping system to take care of automated responders.
Or some random real movies that contain a message: CLICK ON THIS MOVIE
jaco
Posts: 18
Joined: Fri May 26, 2006 5:40 pm
Location: Chongqing, China

Re: Scoring method

Post by jaco »

Sasari I hope you are wrong about abusing the system, but I think you're right, the cheating has begun. It's very sad indeed.
juco12
Posts: 3
Joined: Wed Aug 02, 2006 9:43 am
Location: Scotland, UK

Post by juco12 »

if you look at some participants rate of scoring you get suspicious..
Well until I read this post I wasnt aware of how to abuse and create higher scores not that I would want to anyway. But I am sitting at a genuine No6 and that would I am sure have been higher if the site had not `shut down`. At one point I got up to No 2 before access was denied....Damn!
Anyhow I am just just enjoying the fun of it, and the children are enjoying taking the P... at me, mind you the fact I have now declared myself a scientist its no wonder :-)

regards
Juco
MaximVesuvius
Posts: 20
Joined: Wed May 17, 2006 8:11 pm
Location: Boise, ID, USA, North America, Earth, Solar System, Milky Way

Post by MaximVesuvius »

I think that the admins will be able to easily see who's doing the the testing movies only. I remember seing a link that shows images that are in "debate", given this I believe that the results recieved are under review.
bocomo
Posts: 2
Joined: Tue Aug 01, 2006 6:52 am
Location: Columbia Missouri USA

Post by bocomo »

Cheating is no fun. The actual thrill for me is hoping to find a track in a real movie!
Zsinj
Posts: 60
Joined: Tue Aug 01, 2006 10:02 am

Post by Zsinj »

Cheating is pointless. What do you have to gain? "Oh look, I'm #1!" but you know what that means to those of us who actually care about the project? It means that our results are not reliable and it makes more work all around.

Do you want to be #1, or do you want to make the people working to make this project heppen hate their lives and contract out the project? If you must cheat, you're not having fun.
goesser
Posts: 6
Joined: Thu Aug 03, 2006 2:42 am

Post by goesser »

It's annoying to see people having several hundred points but you don't know if they have cheated.
(via url: next_movie_type=..&ondeck_movie_type=...)

Some people say "I do it for fun only" but only for fun I can do other things as well.

I won't blame anyone but anyway, it discourages me.

:(
Whity
Posts: 23
Joined: Wed Jun 28, 2006 4:29 am

Post by Whity »

It can be quite frustrating to people who do the real job at the best possible rate they can and still see people going twice or triple as fast. This makes people worry about quality of work. For example, someone with 800 points, and 1 cali movie in 4 means he has waited for 3200 movies to load, scrolled to each of their focus levels, analyzed them thoroughly and made an educated decision about wether there is a track, without making any errors in calibration movies..

If we take like 20 seconds for a full movie analysis, which is basically the amount you need to get everything loaded and analyzed in your brain.

That means 17 hours straight non-stop working... No human being can do that without error.
I wouldn't mind for the ranking system to be reviewed and after change of scoring method a reset would be cool.

Just my 2 cents.
Zsinj
Posts: 60
Joined: Tue Aug 01, 2006 10:02 am

Post by Zsinj »

I think the "score" formula needs to be changes. Have all five key values (correct cali, incorrect cali, total real, specificity, sensitivity) come into play.

Something along the lines of:

a=(spec+sens/2)
b=(correct-incorrect)*a
score=b+(a*real)

This way, regardless of ratio, the amount of real movies you have done correctly (correctness tested using the calibration movies) comes into play.
Join us on the Unofficial IRC channel!
irc://irc.freenode.net/stardust@home
Orion_0169
Posts: 153
Joined: Wed Aug 02, 2006 8:17 pm

Post by Orion_0169 »

Whity wrote:If we take like 20 seconds for a full movie analysis, which is basically the amount you need to get everything loaded and analyzed in your brain.
And if someone is twice as fast as that fabricated value, your 17 hours becomes 8.5. And if they are three times as fast ast the fabricated value, then it's under 6 hours. Once again, maybe some people ARE just better at it than others.

Sounds like there are some issues (I for one haven't been paying attention to URLs in the address bars, I'm watching the movies) but I hope that the system doesn't get tweaked to take away from more efficient participants' progress - just because of some whiners.
Je ne peux pas regarder la Mer sans me demander qui vit au-delà de cela.
Zsinj
Posts: 60
Joined: Tue Aug 01, 2006 10:02 am

Post by Zsinj »

Orion_0169 wrote:Sounds like there are some issues (I for one haven't been paying attention to URLs in the address bars, I'm watching the movies) but I hope that the system doesn't get tweaked to take away from more efficient participants' progress - just because of some whiners.
No need to get defensive. ;) We all need to remember that people work faster than others and can work fewer or more hours than we do ourselves. While new leaders may be shocking now, let the project evolve for a week or two and I doubt we'll see runaway leads any more. :D
Join us on the Unofficial IRC channel!
irc://irc.freenode.net/stardust@home
Orion_0169
Posts: 153
Joined: Wed Aug 02, 2006 8:17 pm

Post by Orion_0169 »

lol Not defensive. My integrity stands. :wink:

Actually, I've been paying more attention to the time involved. And the 20 seconds average gets absolutely blown away by many of the obvious callibration movies. Many of those I have done in a second or less. Often, if you start the focus at the bottom, you'll see the track for the callibration pop up right away. A quick check in the focus and you're done in no time. Doing a one or two second movie one in four times... well, it explains how people can be doing well.

Like I said, praise the people who are actually trying to help.

And like they say, if you're not having fun...

:-)
Je ne peux pas regarder la Mer sans me demander qui vit au-delà de cela.
M.B. Frederiksen
Posts: 3
Joined: Fri Aug 04, 2006 2:48 am
Location: Denmark

Sit back and relax

Post by M.B. Frederiksen »

It seems to me that a LOT of people here are concentrating about "possible cheaters" or other thoughts regarding the scoring system.

To my knowledge the scores is of NO importance to the project.
What counts is how many people identifies a real particle track and submits this finding to the scientists for further investigation.

Image
A particle caught in the Aerogel
"Even if you are on the right track, you will be run over if you don't move"
templar781
Posts: 38
Joined: Thu Aug 03, 2006 9:19 am
Location: St. Petersburg Florida
Contact:

Re: Scoring method

Post by templar781 »

Sasari wrote:The current scoring method is blatently open for major abuse. If you see ondeck_movie_type=testing in the link you know it is a test video. You can spend your time looking for a track. If it says real you are safe to just click no track.

Easily gaining a lot of points. I'm sure you could get even more creative with how the links work to give yourself testing movies on purpose. I suggest a system that doesn't let the user directly know if its a test video or real data.
It looks like the scoring system has been turned off. My rank this morning was 597 and it's still the same after going through approx 200 movies. I would think it would have moved at least a little bit.
Last edited by templar781 on Fri Aug 04, 2006 7:39 am, edited 1 time in total.
Winning isn't everything, but wanting to win is. - Vince Lombardi
Post Reply