Questions on Ranking ...

Discuss your experiences with and ideas about Stardust@home here.

Moderators: Stardust@home Team, DustMods

CerealKiller
Posts: 12
Joined: Tue Sep 26, 2006 12:49 pm

Post by CerealKiller »

Every movie processed is logged with name and parameters, so I can review later to see what were the problems. I can give a list of the types of problems, but ...

My first try was "attacked" because of the fact that no one knows exactly how the real tracks look like and the program did search for features seen in the training and calibration movies. As I posted previously this program does not look for those "track like" features. It searches for sharp visible features under the aerogel surface (objects that can be seen only under surface or clearly go under surface and are not blurred surface features).
So to answer Nikita, it finds "every" kind of track or object that seems to be "inside" the aerogel.
Bad focus movies are a large container for various kind of bad movies. The program can identify out of focus movies. It determines surface position and tilt, which is approximated by a plane in 3D. I can give good rational on the choice of plane and I am aware of it's vulnerability too.
Thanks to the Low Angle Tracks thread, I was able to test it also on those and I found the results satisfactory.

If you are interested in the stats of the first program, they can be found in the old posts in this thread.

Where I wrote idp I meant "possible interstellar dust particle". Thanks.
Nikita
DustMod
Posts: 994
Joined: Wed May 17, 2006 8:33 pm
Location: Indiana, USA

Post by Nikita »

Wow! Can you write me a program that helps my family identify dust in the house and how to remove it??? (Just kidding) Amazing job!
From dust we come
fjgiie
DustMod
Posts: 1253
Joined: Sat May 20, 2006 8:47 am
Location: Hampton, SC, US

Post by fjgiie »

Hi CK/SkyNetV1,

1. Did SN have trouble with the weak small cals?

2. Is movie 5262416V1 in SN's Events list? Small possible track at 377, 264.
http://stardustathome.ssl.berkeley.edu/ ... =5262416V1
(Log in then hit the back button twice, weak track-look at coords=? 377, 264)

3. My events with no duplications:
Real movies 157,458
305 different movies from 347 clicks. 0.19%, 305/157458
71 cut 1 for 23%, 71/305
54 IS for 18%, 54/305

SN's cut 1's compared to movies viewed is 0.068%, 196/287499
mine is 0.045%, 71/157458 - I stopped clicking what I thought were inclusions early.

Thanks,

fjgiie
CerealKiller
Posts: 12
Joined: Tue Sep 26, 2006 12:49 pm

Post by CerealKiller »

There are several type of problems:
- Movies with strange and complex surfaces.
- Movies with significant drift. (as if the microscope would be moving relative to the gel and I do not mean tilted surfaces)
- Movies that do not contain features, only a track. (a few surface objects help to find tracks)
- Track CM's that are not fully correct. (with no visible surface or features a track appears)
- My favorite No Track CM that contains 2 tracks. (or that seems to be tracks)
- Stubborn Amazon server sometimes not serving specific frames.

(I could post examples, but I only have the file name of the movies, not the identifier that the event viewer uses. Maybe someone knows the transformation between them?)

Thanks for the movie.
It is not in the event list, so I checked why not. The program would click exactly on the track, but then it cancels the click, because it finds the feature too close to the surface (only two frames under surface, one more frame and it would be valid). I adjusted one of the 4 decision parameters a bit, so now it finds it a valid track.
After I calibrated the parameters for several hundred tracks (and no tracks), these on-the-edge movies help me to set the parameters very precisely.
oaklea
Posts: 11
Joined: Tue Jan 30, 2007 9:29 pm
Location: New Orleans, LA, USA

Multiple objects below the surface

Post by oaklea »

There are movies that have several objects (presumably inclusions) below the surface that come into focus at about the same frame as well as those where the multiple objects come into focus at different frames but still well below the surface.

What does do in these cases? I don't have any of these movie IDs handy right now, but I could probably find a couple if it helps to indentify them.
jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: Multiple objects below the surface

Post by jsmaje »

oaklea wrote:There are movies that have several objects (presumably inclusions) below the surface that come into focus at about the same frame as well as those where the multiple objects come into focus at different frames but still well below the surface.

What does do in these cases? I don't have any of these movie IDs handy right now, but I could probably find a couple if it helps to indentify them.
See GelDelve's useful comments here. If you are a 'conservative', and are convinced they are all inclusions, then you should probably click No Track. If, on the other hand, you are a 'liberal' then simply click on one of them (doesn't matter which) and the movie will get looked at - eventually. Personally I've found myself swaying between one approach and the other depending on unaccountable moods, though I tend toward the 'liberal'.

Be assured that our individual vagaries are extremely unlikely to be critical, one of the greatest strengths of this distributed project.

PS oaklea: not sure that 'Multiple objects below the surface' is relevant to this particular 'Questions on Ranking...' thread.
oaklea
Posts: 11
Joined: Tue Jan 30, 2007 9:29 pm
Location: New Orleans, LA, USA

You misunderstood my question

Post by oaklea »

I was asking how SkyNetV1 handled these movies.

Oak
jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Re: You misunderstood my question

Post by jsmaje »

oaklea wrote:I was asking how SkyNetV1 handled these movies
Sorry Oak, you're right that I misunderstood your question: it's because your post simply said "What does do in these cases?".
I presumed you actually meant 'What does one do...' (i.e. you, I, everyone else), rather than 'What does it do...' (i.e. SkyNetV1).
I'll be as interested as you to see what CK has to say.
fflo
Posts: 63
Joined: Tue Aug 08, 2006 7:37 am
Location: Eindhoven, Netherlands

Questions on Ranking

Post by fflo »

CerealKiller wrote: My first try was "attacked" because ..................................
Allow me to comment on "attacked" : The meaning must be that your skillful work provoked comments from our pseudo-scientific community and - as usual in science - at least they did not deter you from bringing us a next step in the right direction !
And again thanks for letting us share your thoughts behind the shaping of your renewed program. As soon as you can sell this (or your next) remarkable program to the team in Berkeley, we shall be relieved from our tedious work, and they must pay for it.

fijgiie woke up some curiosity in me to these figures:
SN's cut 1's compared to movies viewed is 0.068%, 196/287499
mine is 0.045%, 71/157458 - I stopped clicking what I thought were inclusions early.
I am , of course , slow compared with him and your (cerealkiller's) programs: rank 120 as of today.
Yet, my figures suggest that a slower pace does not necessarily mean a lower achievement:
Total real movies viewed 33640
total events 483
possible idp's 70
(not counting "passed cut 1 but need rescan")

So, fijgiie, this is a hitrate of 70/33640 = 0.208% !
For the peace of mind of all the consciously slow dusters, in Dutch we say:
"Haastige spoed, is zelden goed", i.e. "Hurry-scurry seldomly does any good". Since this is old fahioned, it does, of course, encompass only us humans, not our number-crunchers.
As long as cerealkiller is working on his program, we have to keep dusting - happily
yours
fflo
fjgiie
DustMod
Posts: 1253
Joined: Sat May 20, 2006 8:47 am
Location: Hampton, SC, US

StardustS2 or CK3

Post by fjgiie »

Just in my opinion StardustS2 or CK3 should evaluate 100% real movies and be allowed to disregard calibration movies all together. After StardustS2 (side 2) reaches a score of around 60,000, the team should freeze the quality rankings of Sensitivity and Specificity at their last levels and freeze the score at 60,000 and allow SS2 to continue on with only real movie evaluations. CK is to be congratulated for working so hard and doing an excellent job with an impossible task.

As for fflo, you are doing an excellent job with a rank of 120 and you cannot consider yourself slow. Your hit rate of 0.208% on cut ones shows what hard work can achieve and you are to be congratulated.

fflo: Ik heb nog steeds één kleinkind meer dan jij.

fijgiie
fflo
Posts: 63
Joined: Tue Aug 08, 2006 7:37 am
Location: Eindhoven, Netherlands

Re: StardustS2 or CK3

Post by fflo »

fjgiie wrote:
fflo: Ik heb nog steeds één kleinkind meer dan jij.

fijgiie
There is always "master above master" : Despite your lower "hit-rate" your hard work will keep you ahead of me - with ranking and grandchildren!

From dust we came - for dust we search - to dust we go
Happy dusting

fflo
jsmaje
Posts: 616
Joined: Tue Aug 15, 2006 8:39 am
Location: Manchester UK

Post by jsmaje »

Does anyone know who/what wonderdogxyz (no.1 for a long time) is/are?

And why is it that the majority in the Top 100 don't contribute to the forums; you'd hope they'd have a lot of accumulated wisdom to pass on...

Just asking :roll: !
Nikita
DustMod
Posts: 994
Joined: Wed May 17, 2006 8:33 pm
Location: Indiana, USA

Post by Nikita »

The don't contribute because they are busy dusting!!! :D
From dust we come
greuti
Posts: 200
Joined: Fri Aug 11, 2006 10:11 am
Location: Switzerland

Post by greuti »

jsmaje wrote:And why is it that the majority in the Top 100 don't contribute to the forums; you'd hope they'd have a lot of accumulated wisdom to pass on...
Accumulated wisdom? You would rather think they could show horny hands, dark circles around the eyes and very detailed photofit pictures of all calibration movies. :wink:

What I would really like to get too is insight into event lists from those high scored dusters.
the moon
Posts: 177
Joined: Thu Aug 03, 2006 12:34 am

Post by the moon »

Personally I fear that many of the top 100 people are treating this more like a video game and losing sight of the real goal. They burn thru as many movies as possible and click the calibration tracks to get the high score, while of course keeping an eye out for a real track in the hopes they'll be first and "win" the grand prize.

This is all speculation, I have no evidence to support it, but here's my reasoning. People who are genuinely interested in the project and want it to succeed know that there is no need for people to be viewing as many movies as they are. Statistically speaking, I'd guess that after about 20 views on a movie the ratio of track/no track votes is not going to change, and yet many movies are viewed over 300 times.

So what's the motivation to sit for days and click thru 100,000 movies? At some point the people interested in the science are going to say "well that's enough dusting for now, there's no need to keep voting on movies viewed over 300 times", so the people left voting are doing it for no other reason but to see their ranking move up. And I'll bet almost everyone in the top 100 has accuracy better then 99%. They must be very good at spotting that single track by now, but would they notice a real track if it looked even slightly different? Their votes are burying the votes from people that study the information on the forum and take their time viewing but may not have time or be interesting in viewing more then an hour a week.

To obtain better results, the team could limit the number of movies you could vote on. Like 100 a week or something. Just because someone dedicates more time doesn't mean their opinion is more likely to be correct. This also would encourage more casual users to join in because it would be possible for them to be the first to view and identify tracks.

(note to the team: you can't implement this now, it would anger too many people that worked hard to get high rankings. You could do it in secret though. See if your data changes significantly if you throw out all votes from everyone after their first 1000 votes.)

By the way, if you're in the top 100 and reading this, then nothing I've said applies to you, thank you for reading the forum. :wink:
And full discloser, my score is 1460, ranked 979
specificity: 99.59%
sensitivity: 99.47%
Post Reply