Questions on Ranking ...

[Nikita]

See Bryan's comments here:
Update: 16 Oct 06...

The computer programs are welcome, but do not replace us. Until we know the actual stats on this program, we can't say if it is good or bad. First we must acknowledge the effort and knowledge it took to create this program and if it is working, that is better for the team.

Bryan has said that they are willing to create a seperate ranking system for the computers, if needed. Since CK quit, there has not been a need to do so. I don't know if I want them to take the time and energy to create a seperate ranking if there are only one or two. If known, they can be identified as we go.

However, computers can only identify what they are told to look for, they cannot identify the abstract and discriminate as we can, unless told EXACTALLY what to do. Therefore, we are still the best searchers out there.

It will be interesting to see what the results are for the computer programs.

[DustBuster]

It appears as though there's another program besides 'Cerealkiller' posing as a Stardust search volunteer ['SkyNetV4'], able to 'search' 24-7. It's too bad it doesn't realize that, while it may be very prolific @ running up high totals of identified calibration movies [out-doing real people], it will hardly be able to find any stardust. This would no doubt be reflected in it's list of data viewed and it's Specificity/Sensitivity %'s. I don't think such 'volunteers' should be juxtaposed with real people in the Community Rankings.
Let's not lose sight of the purpose of the project. L8R, Crumplestiltskin
_________________
SPACE IS THE PLACE !

Though "SkyNetV4" has risen through the ranks at break-neck speed, they have not yet come forward and publicly identified themselves as an automated program. There are other explanations, such as a group or groups of users using the same login- in any case, the S@H team is aware of their rapid accumulation of points.
I would point out that we haven't had any indication that automated programs are not allowed. This is discovery science and I would include any tools created to facilitate the process as additional discovery science. It's not only amazing, but also inspirational the number of talented people on this planet that are adding their collective knowledge and skills to this project.

We did see that CerealKiller had it's limitations, but it was also a tribute to the tenacity of human resolve; We were told it could not be done- that statement has ever been the motivation to make us (humans) try even harder to overcome an obstacle or limitation.
The fact is that the input from a single user, human or otherwise, will not affect the overall results of the data (genious!) and the development of these types of tools has great potential benefit.

To view the ranking as a reflection of value to the project could be disappointing... and would be absolutely WRONG! To selflessly contribute your time and effort to the expansion of our understanding and knowledge of who, what, when, where and why we are- propels you to the rank of HUMAN:First Class; which is the rank held by every person who has made/will make a positive contribution to this project.

[CerealKiller]

Hi!

I just read some posts on the different forums.
SkyNetV4 is a program.

CK.

[DustBuster]

Other than this forum?

[CerealKiller]

Sorry. I meant different threads.

[jsmaje]

So CK, SkyNet4 is another program. How do you know - written by yourself perhaps (if not, someone you know)?
I really think it's time we were told, since this issue has had quite an effect on the morale of the worldwide volunteer participants doing it all by eye and hand in their own (maybe scarce) spare time.

The worrisome question has been whether such programs as yours are simply identifying positive calibration movies (compressed jpeg pixellation plus big black circular tracks etc.), and gaining high scores just by number-crunching, meanwhile ignoring the various more subtle features to be found in 'real' movies.

I may be quite wrong, but just what were/are the figures for CK's/SkyNet4's sensitivity & specificity, and particularly the length and content of their Events list, and how many have passed Cut 1? This would demonstrate how many features of a quite subtle nature (as it has so far turned out anything of real interest seems to be) have been detected by such programs.

To be honest, I (as a total amateur) have started messing around for fun with Fourier/Hartley transform approaches to seeing how easy it may or may not be to pick out pixellated jpeg patterns, and am finding it very educational. But if you have some deeper secrets, ? neural network pattern-recognition or whatever, why not share it with us?

The point is that only if one can be sure such methods can regularly and reliably find features of real interest as well as or better then the eye and brain will it be useful to apply to a project such as this. I think I speak for everyone involved in wanting to know more. Convince me with enough evidence and I may well believe...!

[CerealKiller]

SkyNetV4 is my next try after the CerealKiller program.
SkyNetV4's core is completely rewritten from scratch.

Current stat:

Spec.: 98.34%
Sens.: 98.30%
Real movies: 287499

7938 movie clicked. (counting each movie once)
Passed 1st cut: 196
Possible idp : 138

As I mentioned, I made a new core algorithm based on mathematics and not on intuition.
It uses 20 frames out of the 40.
Estimates aerogel surface.
Clicks every movie that has a sharp visible object (that is round to some degree) under surface.

Outline of the algorithm:
For every block (8x8 pixel) in every frame determine if there is a significant variation in color.
For every significant block in every frame determine a block sharpness value.
Using the frame numbers of the maximum sharpness values (without outliers) a plane is fitted using least squares method. (This is the aerogel surface)
The block sharpness values under surface are transformed and filtered.
The location of the maximum of the filtered values is the potential track.
Using morphological operators the potential track and its vicinity is processed.
4 decision parameters are calculated to decide whether it is a track or not.

Currently I think one can not perform any (useful) image processing including more than one block (8x8 pixel), because the gradient between the blocks is way too big.

If anyone is interested in details, I am open to discuss.

[GelDelve]

Hip hip hooray for CK!

I think this is a real go and I'm glad you have created a program that is more realistic and accurate than your original program was.

No one can know for certain whether their contribution has been worthwhile, including yours CK, but I'd like to believe that if I knew what you know about programs and the internet, I'd be doing the same thing. Yes, I know I would. I don't in any way think it is cheating or dishonest. I think it is one person's dream to accomplish what others wanted to dismiss as impossible.

I'm still trying to find a way to trisect an angle at age 60, only because it was proved impossible!

[Nikita]

For those of us with no programming ability whatsoever, I congradulate you on such a feat! I hope it does work and helps us find the stardust! Please let us know how it continues to work, we are all very curious! By the way, is it stuck on those tricky calibration movies, or can you tell?

[cthiker]

I think this is a real go and I'm glad you have created a program that is more realistic and accurate than your original program was.

Could not agree more! I've just been a passive observer of this thread, but as someone who has a science and IT background, I heartily congratulate CK for his bold efforts!! Of course, this begs the next round of development - your algorithm is sound to the point that it can be at this time. However, when (and by whomever) the first true particle is identified, it would be nice to know so that adjustment and refinement can be made to that algorithm. Further, if we could get several of these "real" tracks identified, possibly working in a Neural Network in order to "learn" what a track really looks like could be invaluable (in terms of time spent searching, human or otherwise).

Nonetheless, since this doesn't "lessen the science" and it is, itself, a developmental adventure (as others have noted), I think CK's approach is noteworthy, and duly appreciated on my part!!

[Nikita]

It should be remembered that the stardust will be different! Therefore, even if one is found and the program adjusted to identify based on that info, it may not be able to identify other tracks that are very different, lateral tracks, for example. Or am I wrong CK???
What intrigues me is how the program can identify irregular surfaces and bad focus movies. How does it know when to determine that? Lastly, what does it think of our little space spiders???
And in the advancement of science, this project has brought a development that is not learned from the dust but here on our own rock in space! Perhaps this idea can lead to more developments in other areas, for example, by another volunteer seeing the program as a possibility in his/her own career?

[cthiker]

...even if one is found and the program adjusted to identify based on that info, it may not be able to identify other tracks that are very different, lateral tracks, for example.

Nikita,
Exactly! Even if a Neural Network approach were taken, we can only "learn" from what we observe, and our error rate will be commensurate with limits to our observations. Humans have similar difficulties, but their complex, massively parallel multiprocessor (MPMP, or our "brain") is wired to manage that quite well - IMHO, thanks entirely to millions of years of evolution! See an interesting article in the NYTimes that deals with the brain's recognition capabilities at...

http://www.nytimes.com/2007/02/13/healt ... 0&emc=eta1

The good news is that, given it's only a "standard" track that this software, or others like it, identify, it still gives us a leg-up on indentification of at least some of the family members, even if not the whole clan. The other good news is that this can be looked at as pre-development of a particle track identifier that could be used on many other missions that collect 'stuff' in various other locations (Oort Cloud, Interstellar media, or even a fly-by of something more local like the contents of Saturn's rings or its spouting moon Enceladus). Makes the aerogel "net" a much more productive tool than it is today. Indeed, there may be applications not yet even thought of by any MPMP as yet!

Thoughts...(anyone)?

Thanks!!!

[oaklea]

I'd be interested in seeing how SkyNetV4's stats compare to others in the top 5 or 10 spots.

I think that having 196 passing cut 1 and 138 possibles is pretty impressive, but I'm curious how the total of 7938 clicks rates with others that have looked at about the same number of movies.

Looking at the stats it seems like about 2.7% of the movies viewed get clicked and about 2.5% of the movies clicked pass cut 1 (realizing of course, that the team has only reviewed some of them, so this number might not mean much).

While my Real movie number is only about 13,300, if I do the same math; I click about 1.1% (157) of the movies and about 7.6% (12 with 10 possibles) of those have passed cut 1 and about 6.3% (10) are probable inclusions.

Again, it hard to use the "Passed Cut 1" as a measure because we just don't know how many have been reviewed from each list.

SkyNetV4 has a Spec of 98.34% and a Sens of 98.30%.
I have a Spec of 99.73% and a Sens of 97.36% (this is low from missing the very small tracks when I first started.)

Is it possible for someone to take a look at the stats of some others that are closer to the same number of viewings and let us know how this all compares?


Oak

[jsmaje]

Thanks CerealKiller for sharing that reassuring info about SkyNetV4: it's clearly not simply targeting positive CMs as was one major concern, and your algorithm makes perfect sense.

As a matter of interest, what were CK's figures in the end, and is the new program demonstrably better?

I do, however, feel bound to compare my own latest results (as has oaklea, above):
Real movies viewed: 37,301
Movies clicked: 583
Passed cut 1: 114 (of which 'possible interstellar dust particle': 86)*
Specificity: 99.85%
Sensitivity: 99.91%
Score: 13128

SkyNetV4 has therefore clicked on 7938/287499 = 2.8% real movies vs. my 583/37301 = 1.6%, yet has found only 196/7938 = 2.5% considered worthy of passing cut 1, vs. my 114/583 = 19.6% (more than 8 times more discriminating). So, I feel safe... at least for the moment!

That said, despite my sometimes sceptical tone and taking into account all other comments above, I am in fact all for your efforts, even if the team have been proven right that your task is a non-trivial one.

So, keep up the good work - at least they're not having to pay you (or are they? )

* Where you said 'ídp' I think you may have meant 'interstellar particle'; they seem to have reserved the term 'idp' in the Events list for interplanetary dust particles.

[oaklea]

Thanks jsmaje for posting your stats. It begins to answer my questions.

I just wonder if the low percentage for SkyNetV4's "Passed Cut 1" is because so many movies have been viewed and the number of those that have been reviewed is low. A few responses from the top 10 should shed some light on that.

I also re-read my post and thought it might have an unintended negative tone.

I'd like to say that I'm 100% in favor of both CerealKiller and SkyNetV4. I'm just interested in the accuracy of the programs. I'd like to see CK's stats as well to see what improvement there's been in the stats for SkyNetV4.

CerealKiller, keep at it. I think it's pretty impressive so far.

Oak