At 01:37 PM 12/07/00 -0400, Eli wrote:
>Joe McMoneagle wrote:
>>
>> The target was later independently judged by a person who was not
>> participant to the experiment, producing a significant effect size
>> of 0.710 +/- 0.289 (p=0.007).
>
>How on Earth do they judge the significant effect size without a control
>group? Did they try having the judge check the guy's drawings against a
>randomly selected photograph as well as the target? Not as far as I can
>tell. If so, the whole experiment is entirely meaningless.
Well, not *entirely*. We can still make informed but informal assessments
of apparently gobsmackingly unlikely phenomena, assuming that the report is
trustworthy. If it's not, there's no point discussing the topic - or any
topic making unusual claims.
But of course I agree with Eli. The significance estimates are based on
procedures usually found only in the technical papers, not on informal web
sites. Here's my own recent exchange with Joe McMoneagle, raising the same
issue:
==========================
>My lifetime [23 year
>record] for good guesses in free-response RV type tasking is about 55% --
>which I figure is pretty good guessing.
How should we calibrate this, Joe? Against what mean chance expectation?
I assume by `free-response' you mean that you could have generated *any*
description of *anything*, which makes your results seem incalculably
impressive.
But I suppose in reality, even with such a protocol, there are only so many
likely descriptors - you're not, presumably, about to describe a sky made
of purple rock, or a giant green grasshopper with Elvis's head. I would
hope your success in a given task is therefore assessed by blind judging
against a small random selection of your previous RV reports, or against a
set provided by others. (The former method might have the advantage of
tapping your idiosyncratic image bank, which might help compensate for
personal response biases.)
=================================
But by all means, let's not look through the telescope. He might be lying,
and his claims fly in the face of... of... Oh, that's right, we don't
actually have a Holy Book in science, just a series of mutable if robust
hypotheses within paradigmatic frameworks, and observations of varying
degrees of reliability and salience.
Damien Broderick
This archive was generated by hypermail 2b29 : Mon Oct 02 2000 - 17:34:27 MDT