From: turpin@cs.utexas.edu (Russell Turpin)
Subject: Re: Science and methodology (was: Homeopathy ... tradition?)
Organization: CS Dept, University of Texas at Austin
Lines: 123
Distribution: inet
NNTP-Posting-Host: im4u.cs.utexas.edu
Summary: More in response to Lee Lady.
Keywords: science   errors   Turpin   NLP

-*----
I think that part of the problem is that I have proposed a
definition of science that I intended to be interpreted broadly
and that Lee Lady has interpreted fairly narrowly.  My definition
is this: Science is the investigation of the empirical that avoids
mistakes in reasoning and methodology discovered from previous
work.  Lee Lady writes:

> I don't think that science should be defined in a way that some 
> of the activities that lead to really important science --- namely
> thinking and informal exploration --- are not recognized as
> scientific work.  

Unless one classifies "thinking and informal exploration" as a
mistake, they fall under my definition.  I hope no one would
consider speculation, thinking, and informal exploration as
unscientific.  

In article <C5nAsF.MH7@news.Hawaii.Edu> lady@uhunix.uhcc.Hawaii.Edu (Lee Lady) writes:
> Seriously, I'm not sure whether I misjudged you or not, in one respect.  
> I still have a major problem, though, with your insistence that science 
> is mainly about avoiding mistakes. ...

Here is where I think we are talking at cross-purposes.  It is not
clear to me that the kind of definition I have proposed should be
taken as describing what "science is mainly about."  Consider,
for example, a definition of invertebrates as all animals lacking
a backbone.  This fairly tells what is an invertebrate and
what is not an invertebrate, but it hardly tells you what
invertebrates are all about.  One can read this definition and
still not know that 95% of all animal species are invertebrates,
that invertebrates possess a remarkably broad range of form, that
some invertebrate groups -- such as insects and nematodes -- are
ubiquitous in all ecosystems, etc.  In short, knowing the
definition of invertebrates does *not* tell one what they are
"mainly about."

The misunderstanding here is my fault.  I did not give sufficient
context for people to understand my proposed definition.

> Okay, so let's see if we agree on this: FIRST of all, there are degrees 
> of certainty.  It might be appropriate, for instance, to demand carefully 
> controlled trials before we accept as absolute scientific truth (to the 
> extent that there is any such thing) the effectiveness of a certain 
> treatment. On the other hand, highly favorable clinical experience, even 
> if uncontrolled, can be adequate to justify a *preliminary* judgement that
> a treatment is useful. ...
>
> SECONDLY, it makes sense to be more tolerant in our standards of 
> evidence for a pronounced effect than for one that is marginal.  

I agree on both counts.  As an example of the second, it would only
take a few cases of curing rabies to convince most veterinarians
that a treatment was effective, despite a lack of controls.  

As to the first, I do not think it is useful to talk about
"absolute scientific truth."  I think it is more useful to talk
about the kinds of evidence that various claims have and the
kinds of evidence IN PARTICULAR FIELDS that in the past have
proven faulty or reliable.  The latter is obviously a matter of
degree, and in each field, practitioners try to discover the
relevance of different kinds of evidence.  

One of the primary mistakes that marks the advocacy of an idea as
psuedo-science is that the advocacy lacks any sense of proportion
regarding the kinds of evidence related to the proposed claim,
the kinds of evidence that are actually relevant to it, and the
historical reasons in the field that certain kinds of evidence
are given more weight than others.  It is perfectly alright to
speculate.  I have read quite a few refereed papers that
speculated left and right.  But the authors were careful to
identify the notions as speculative, to list what little evidence
was presently available for them, and to describe how research
could proceed to either put the notion on more firm footing or to
uncover its problems.  Often what distinguishes whether a paper
of this sort passes muster is the thoughtfulness with which the
author sets the context and paves the way for future work.  (It
is in this area that many proponents of speculative ideas fail.)

> The folks over in sci.psychology have a hundred and one excuses not to
> make this simple test.  They claim that only an elaborate outcome study
> will be satisfactory --- a study of the sort that NLP practitioners, 
> many of whom make a barely marginal living from their practice, can ill 
> afford to do.  (Most of them are also just plain not interested, because 
> the whole idea seems frivolous.  And since they're not part of the
> scientific establishment, they have no tangible rewards to gain 
> from scientific acceptance.) 

I think a lot of scientists steer away from things that --
deserving or not -- garner a patina of kookiness.  When
proponents of some practice see no value in more careful
investigation of that practice, that sets alarms ringing in many
researchers' minds.  

This is unfortunate, because there is undoubtedly some
intersection between things that are worth investigating and
things that are advocated by those who seem careless or
unreasonable in their advocacy.  On the other hand, I can
understand why many scientists would just as soon select other
directions for research.  As Gordon Banks has pointed out, no one
wants to become this generation's Rhine.

> One academic in sci.psychology said that it would be completely 
> unscientific for him to test the phobia cure since it hasn't 
> been described in a scientific journal. ...

I think this is absurd.  

> Actually, at least one fairly careful academic study has been done 
> (with favorable results), but it's apparently not acceptable because
> it's a doctoral dissertation and not published in a refereed journal.

I wonder why the results were not published.  In my field,
dissertation results are typically summarized in papers that are
submitted to journals.  Often the papers are accepted for
publication before the dissertation is finished.  (This certainly
eases one's defense.)

Finally, I hope Lee Lady will forgive me from commenting either
on NLP or the discussion of it in sci.psychology.  I know little
about either and so have nothing to offer.

Russell
