Is The Science In Jeopardy?

This forum is for discussing space science topics related to Stardust@home.

Moderator: DustMods

Starzack
Posts: 1
Joined: Thu Aug 03, 2006 1:38 pm

Is The Science In Jeopardy?

Post by Starzack »

Unless it can be shown that cheating is impossible or that it would have negligible impact upon the science results, I feel that the program should be shut down until the issue of cheating is properly addressed.

As has been pointed out already, it is possible to know that the movie you are looking at is a real one or a test just by reading the URL.

Furthermore, while doing my own work, I was observing the scores of some top scorers increase in a way that is nearly impossible, or at least highly questionable. In one case, I feel 99% confident that one person was in fact cheating.

Cheating, in the sense of simply going through as many movies as possible to increase their score with little or no regard to the scientific purpose of this endevour.

Early this morning I observed the score of JFM3 move from 556 at 12:16 am to 763 at 01:16 am. That's an increase of 207 in one hour. Based on my own work, it appears that 1/3 of the movies are test images, therefore that means that you view about 3 times as many movies as your score increases. That means that JFM3 saw about 621 movies in one hour. That averages out to one movie every 5.8 seconds. Some statements I found in the FAQ indicate that one in five movies will be a calibration image. If that is true, then that makes JFM3's case even worse, meaning he had to view 1035 movies in an hour requiring a sustained average rate faster than a movie every 3.5 seconds!

Furthermore, between my last look at the scores early this morning and my next look today, I saw several people jump quite dramatically in those 11 hours and 47 minutes.

MadmanNR went from 208 to 1077, requiring a sustained average rate of one movie every 16 seconds over that entire near 12 hour period. (presuming 3x movies viewed to calibration ratio)

MaximVesuvius went from 474 to 1012, a sustained average rate of one movie every 26 seconds.

McFly wasn't even on the chart at first, so his score went from less than 131 to 1031. Using 131 as the mark, that would require a sustained average rate of one movie every 16 seconds.

16 to 26 seconds is not an unreasonable amount of time, but bear in mind that is an AVERAGE rate over nearly 12 hours. Unless these folks can say they were awake and working all morning from a little after midnight pacific time to mid day pacific time, they would have to be working much faster than calculated in order to maintain that long term average score.

My point in all this is not that someone is out scoring me. My concern is that there is obvious cheating and it is obvious there are people working really really fast, and that this cheating and speed is going to jeopardize the ability of this scientific project to meet it's goals.

Science is not a race. Science is a slow painstaking process. One cannot rush through doing something like this.

I feel the program should be stopped until at the very least the issue of being able to tell a calibration from a real movie is eliminated from the URL.

I also feel the scores page should show a persons sensitivity and specificity scores as well. This would show people that those who are scoring quickly may be getting there through sloppy work.
gavin42g
Posts: 31
Joined: Tue Aug 01, 2006 4:08 pm
Location: Kelowna (-ish), BC, Canada

Post by gavin42g »

Frankly, I think the entire "rank" concept should be scrapped. You're absolutely right: science isn't a competition (not fundamentally, at least), and just having a high specificity and sensitivity score should be enough motivation for anyone interested in the project itself to do the best they can. But putting the project on hold for something like this would do more harm than good, in my opinion.
MaximVesuvius
Posts: 20
Joined: Wed May 17, 2006 8:11 pm
Location: Boise, ID, USA, North America, Earth, Solar System, Milky Way

Post by MaximVesuvius »

gavin42g wrote:Frankly, I think the entire "rank" concept should be scrapped. You're absolutely right: science isn't a competition (not fundamentally, at least), and just having a high specificity and sensitivity score should be enough motivation for anyone interested in the project itself to do the best they can. But putting the project on hold for something like this would do more harm than good, in my opinion.
Without any hint of sarcasm I completely second your motion to abolish the ranking system. At least then I can have peace. To the original poster: by your standards, how much time do I have to spend on a movie? 2~5 mins? I have been trained in defect analysis, and if I'm doing wrong I prefer to be told by UC Berkeley, NASA, or the Planetary Society because they have my logs of what I have done yesterday and what good/bad came of it, you have nothing but your own timings (which are not mine). As for how much time I spent yesterday: ~9:30 a.m. to 11:50 p.m MNT, why so much? Cuz I'm enthusiastic about sharing a project that NASA had a hand in (childhood dream), and I'm in summer vacation.
Zsinj
Posts: 60
Joined: Tue Aug 01, 2006 10:02 am

Post by Zsinj »

I agree with Maxim. I have said numerous times that some people work faster than others. Assuming that people are cheating without watching them work, or knowing about their background like in Maxim's case, is assuming bad faith in the project and its process, which I chastise.

Let the project work itself out over the next week or so. New people are coming into the project every day and each of those people work differently.

Original poster: You went through a lot of work to make an attack on the project your first post. If you don't care about rank, why do you care so much about other people's rank? I'm not defending my Top 20 position, but I am saying that you should have made a civil suggestion than a falsely accusing attack.
Join us on the Unofficial IRC channel!
irc://irc.freenode.net/stardust@home
Nikita
DustMod
Posts: 994
Joined: Wed May 17, 2006 8:33 pm
Location: Indiana, USA

Post by Nikita »

Perhaps questions should just be directed strickly to the Stardust team without other volunteer names. This is a worldwide community project and we should all work together. No one should have to defend his/her rank or speed, nor process. This has been brought up several times and I'm sure will be addressed as soon as they get the chance. This is all new and we are scoring many more possible tracks than they thought would be possible, we have had some surprising problems at start up. When we had the major shut down, it was well into the evening (Pacific time) before it was up. So we know that they are working very hard to get this going for us. Perhaps we should all be patient and focus on the films. This will all sort out in time, but it may take a little time. Until then, just have fun dusting!
From dust we come
Merovingian
Posts: 22
Joined: Wed Aug 02, 2006 11:33 am
Location: Juneau, Alaska, USA
Contact:

Post by Merovingian »

I don't like the rank system either, but I agree that we should assume good faith until any cheating really can be proven.
In the dark of night, there lives the lighthouse keeper;
And the lone survivor of this world is he...
Solkos
Posts: 14
Joined: Thu Aug 03, 2006 8:17 pm
Location: Anoka, Minnesota

Post by Solkos »

Cheating doesnt really make a huge impact to be honest. Even if half the top 20 are cheating, the videos are viewed 10-20 times randomly between 4000+ people currently.(not sure of exact numbers) If the 19 others miss a track, especially a big one, than something is seriously wrong.

What is the point of a public ranking system anyways? So you can brag to your buddies that you spent the most time of anyone in the world looking for pieces of dust? :lol: Ok they arent just pieces of dust, they are pieces of dust that will hopefully be named after myself pretty soon!!! :D
Solkos
Posts: 14
Joined: Thu Aug 03, 2006 8:17 pm
Location: Anoka, Minnesota

Post by Solkos »

Ohh and just as a side note, it does take less than 4 seconds on most videos. You find the surface and than you go below it. Finding the surface can be done in about .5 seconds on most of these images. Once you are below the image you focus up and down and check every black image to make sure they are not comming into focus but going out of focus.

With that out of the way, I mark a LOT of these movies as out of focus eather because they have a huge mess on them that can't focus or because they don't focus low enough but it only takes 1 second to realize this.

You can't really take internet connections into account because I am loading images so fast that they are normally ready before I am done with the previous video. Sometimes I can spot tracks in the sample videos before the next is loaded but it's fairly rare. I am only on a cable modem. There are people on faster connections than me like on campus college students or people at work. I only use single windows to load images but throughout the forums I have seen people talking about using 2 windows so they never have to wait for load times.


Calibration Movies Answered Correctly 554
Calibration Movies Answered Incorrectly 6
Your Overall Score: 548
Total Real Movies Viewed: 1190
Your Rank: 28 out of 5417
Specificity: 100%
Sensitivity: 98%
Lomic
Posts: 20
Joined: Wed Aug 02, 2006 4:51 pm

Post by Lomic »

I'm more concerned that a majority aren't paying attention. Check out these two of the sixteen I've flagged so far.

First one has 10 of 18 Agreement:

http://stardustathome.ssl.berkeley.edu/ ... e_id=41876

Second one has 1 of 22 Agreement:

http://stardustathome.ssl.berkeley.edu/ ... e_id=43496

They are the same size, roughly same position and have little or nothing else around them - only difference is the first is slightly darker. Of the rest of mine which all look very similar, only one other has Agreement above a 3. I'm really anxious for them to release *REAL* movies of what they are and aren't looking for, rather than the controlled tutorial movies.
Solkos
Posts: 14
Joined: Thu Aug 03, 2006 8:17 pm
Location: Anoka, Minnesota

Post by Solkos »

Wow I am kind of surprized that it is only 1 of 22 on that one. It is very light but easy to catch in a video that is as un cluttered as that one. I am almost sure that it is not a track though just because it is so deep and has no track leading up to it. The first one just looks like a dark spec also but they are definately 2 that I would flag because ultimately, it is not really my decision to determine if it's a track or not.
Jwb52z
Posts: 61
Joined: Thu Aug 03, 2006 5:05 am

Post by Jwb52z »

My whole opinion on the scoring and specificity and sensitivity is that it shows me just how much I suck at doing this project. I'm only right about 40 percent of the time in the test images. I am starting to wonder if I should bother to continue.
Lomic
Posts: 20
Joined: Wed Aug 02, 2006 4:51 pm

Post by Lomic »

Exactly. I'm certainly not stating I believe these are tracks at this point, but just that people are flagging one and not the other makes me scratch my noggin.
Peter Borah
Posts: 15
Joined: Mon Jul 31, 2006 3:22 pm

Post by Peter Borah »

I like the rank system. I'm currently in 750th place, so I have no bias. It gives me a way to measure progress, and a reason to do more work. I'd have done half of what I've done without a ranking system.

(And I'm 98/98, so I'm not being sloppy.)
greyhorse
Posts: 6
Joined: Mon Jul 31, 2006 9:02 pm
Location: Indiana, USA

Post by greyhorse »

Well, maybe it's because I lack the proper chromosomes, but I don't really care about "rank" or "scores". I think it's interesting, but not something to be concerned about. I do agree that it is a quick process to find a track in a movie, but I still spend "extra" time on the ones that seem pretty obviously track-less. Just my cautious personality, I guess. I know plenty of others will be looking them over, and if I miss something surely plenty of others will catch it- I just can't help fine-tooth-combing the images. Of course I have no training... just a real enthusiasm for being a (very small) part in this process. Why else would I be up in the middle of the night staring at weird blurry blobs on my monitor? :D

Congrats to those of you with the training and/or confidence to quickly work through these things! :)
Malc_UK
Posts: 3
Joined: Thu Aug 03, 2006 1:10 pm

Post by Malc_UK »

What does it matter? There are some of us who take the time to study each movie in detail. to be honest, those who are "cheating" by going through as many as they can in a short as possible time are only cheating themselves. The whole point of this exercise is to find some stardust.. going through images too fast will mean you'll miss them.

I've gone through 100 images.. it's taken me about 2 hours (I work for a living, so I can only though this in my spare time)
however I've already found 2 candidates.. that makes me happier than being on top of the scoreboard ever would :D

Let the "cheaters" get on with it. Like someone already said, this is science, not a race, as long as most people are taking their time and find some tracks, the project will be a success regardless
We are Dyslexia of Borg. Futility is resistant. You will be ****-laminated
Post Reply