discussion

1

Research Methods Lecture on Chapter 11: Ecology of the Experiment

Ecology is the relationships of all the parts of an environment to one another. We typically

associate ecology with the natural environment, our planet, but it can also refer to smaller parts

of our planet, like the little environment of the experiment. This environment is made up of the

researcher, the researcher assistants, the subjects or participates, the apparatus and/or materials

used to conduct the study and gather the data, the

room and surrounding areas during the study, and

even the society, culture, and historical times in

which the study is taking place. Each of these

influence the outcome of the study making it

important to be aware of ecology and to be aware of

the potential bias that may result. If the bias is too

great we may have a problem with Ecological

Validity, the extent to which our study was

influenced by the environmental factors inherent in

all research.

Experimenter Factors

The most consistent principle in the universe is

accident and error. We are full of them. Just because

you decide to do some research does not mean you

now are above accident and error. You are the same error-prone, accident victim, you've always

been. Look at your GPA. It's a measure of your mistakes! A part of your personal equation!

Have you seen the T.V. show about being a millionaire? If you get stumped on a question, you

can call a friend or ask the audience. The audience is right over 90% of the time and the friend is

right about 50% of the time. One way to reduce accident and error is to ask the audience. Go for

the group. The larger the group, the better. As experimenter it's a good idea to find some other

experimenters to help us out. If you are watching video tapes of children's play behavior and

counting instances of pro-social behavior, get some help. The more assistants involved in the

scoring the more accurate your findings. Experimenter effects consider how the results of a

study may have been influenced by the attitude or behavior of the experimenter. Two are

especially important.

Biased Data. An experimenter is not objective. An experimenter nearly always has a preferred

outcome. I want theory X to be supported. This experimenter is designed to show that Theory Y

is baloney. Researchers who study researchers have shown us that the errors that we make

(measurement errors, extraneous variables) most often

add support to our preferred outcomes. These errors are

not random, but are biased. This is often called the

personal equation. If a different researcher had collected

the data, a different outcome may have resulted. Of

course we take great effort in designing experiments to

reduce the effects of the personal equation as much as

possible. More than one research assistant is a good

2

idea. With at least two researchers, an interrater reliability index can be calculated (Cohen's

kappa is the usual statistic for this), which tells us the extent to which the two researchers agree

with one another.

Biased Interpretation. Just as unconscious bias adds favorably to our measurement error, our

preconceived view of the world adds to the way we interpret the findings. In my statistics class

there is a question requiring a z-score analysis. A local group home claims it is better than state

run programs and suggests that locally run programs should replace state programs. The data

indicate that indeed there is a statistically significant difference between the local and state

programs. But the state has the better program! This contradicts the stated claim. Nonetheless,

most students miss this and because they found a statistical difference they conclude the local

program is better. After all, that was the wording of the problem and the hypothesis. If a person

can be so easily duped, so easily led to not see the obvious, how can we honestly accept the

interpretations of data? We cannot. It is all hoax and fraud. The theories are temporary

metaphors, cute stories of phenomena. The data is the real thing. You can't argue with data. But

you can claim it means things it doesn't.

Biasing the Subjects. The experimenter can easily

bias the subjects responses/behavior and can do so

without being aware of it! One's attitude, demeanor,

facial expressions, etc., can provide clues to the

subjects about what behavior the experimenter

expects. Subjects will then behave to fulfill the

expectation, or will behave counter to the expectation.

Additionally, such things as attractiveness, dress, and

gender can influence subjects' behavior. The best way

out of this problem is what is called a Double-Blind

study. The subjects may know what the experiment is

about generally, but should not know which group

they are in (experimental or control); the subjects are

"blind" to the which level of the IV they are getting. The experimenter can also be "blind" about

which group the subjects are in! Thus it becomes a double-blind study and the bias of the

experimenter is now unrelated to the particular group of the subjects and is thus random error,

not systematic error.

3

Reducing Experimenter Bias

There are ways we design studies to help reduce bias. One has been mentioned: use several

research assistants to help collect, analyze, and interpret the data. A second one is to use the

blind or double-blind method. A third way is to simply be open and honest about your bias and

make conscious effort to view your data from a theoretical perspective you don't agree with! I

remember one of my favorite things in debate club in college was arguing for things I personally

disagree with. Stepping into the shoes of a perspective I thought was wrong, yet doing my best

to argue in favor of it taught me more about that position than any attempt to discredit it.

Subject Factors

Almost always, people know when they are subjects of a psychological study. This knowledge

almost always changes the way people behave. The classic example of this is the Hawthorne

Effect, named after the Western Electric company's Hawthorne plant. The study was designed

to find ways to improve factory productivity by changing the factory environment: add music,

change the color of the walls, change jobs on the

assembly line, etc. What was eventually discovered

was that any change in the environment increased

productivity. Your textbook says that the subjects felt

they were "special" since they were selected to

participate in the studies. More likely, I say, is that

they knew the bosses were now looking over their

shoulders, so they better be productive! A more

general truth may be here too. Perhaps after years of

working in the same place, doing the same job, any

change in the workplace is seen as an improvement

and perceived improvements in the workplace produce

improvements in productivity.

A similar finding is discussed in education classes. It's called the Rosenthal Effect. Rosenthal

was an educational researcher and discovered that teacher expectations were the most important

factor in educating the young (even more important than teaching method, supplies, salaries,

classroom setting, etc.). He faked some files on some students so teachers thought the good

students were bad and bad students were good. By tracking the progress of these students and

others he discovered that good students grades declined (teachers thought they were bad and sure

enough they got worse) and bad students improved. The teachers expectation biased their

teaching in such a way to support their expectation, or the students behaved in such a way to

fulfill the teachers expectations (or both). Together, the Hawthorne effect and the Rosenthal

effect bode ill for science. On the one hand, any treatment you impose on the experimental group

is likely to result in improvement and on the other hand, your subjects will pick up on your

expectations and behave accordingly. It's a wonder science is useful at all. You can use the

double blind procedure (see above) to help reduce problems due to expectations, but subjects

almost always know when they are receiving some treatment. Often researchers will employ a

placebo to help reduce the Rosenthal effect. A placebo is a fake version of the real treatment; a

4

sugar pill. You have two groups, the real experimental group and a "placebo control group."

Both groups believe they are getting the treatment, so the Hawthorne effect is in full force for

both groups, while the Rosenthal effect is controlled by the double-blind procedure (since the

experimenter does not know which group is getting the real thing and which is getting the

placebo, the experimenter cannot form expectations that would affect the groups differently).

Thus, any differences between the two groups can only be attributed to the real treatment. If

there is no treatment effect, then both groups may show the same level of "improvement".

Clearly if a sugar pill is as good as the real drug, then the drug is not responsible for

improvement, but it is the belief in the drug that is responsible.

Demand Characteristics

There are many factors to consider in designing and implementing a study. The various factors

that could influence the subject are collectively known as demand characteristics. Some

textbooks refer to this as reactivity because it is the features of the research setting that cause the

subjects to react differently than they normally would. Just being observed may cause subjects to

behave abnormally (like seeing a large mirror in the laboratory room - a one-way observation

window?). In survey research some people habitually answer "no," while others habitually

answer "yes." High School graduates who are going to college typically report that they do not

use and have not tried marijuana, but those not going to

college say they do use marijuana (a 2:1 difference).

Objective measures of marijuana use show us that both

groups are equal in marijuana use. Volunteer subjects

differ from non-volunteers. They are more intelligent,

more educated, more cooperative, better adjusted, and

desire social approval, compared to non-volunteers. It

requires considerable effort to measure the non-

volunteer, those who refused to participate in your study,

or left during the study, but if you don't then your results may not be valid. Magazine surveys,

radio shows, television news programs, web-based surveys, etc., that rely on volunteers

answering questions are not valid.

Subjects often do things just because an "authority" told them too. They even do dangerous

things they would never ordinarily do. One study found that 50% of the subjects would reach in a

cage, grab, and remove a venomous snake, simply because the

experimenter told them to! In the same study, 83% of the subjects

reached in a bucket of acid to remove a coin and another 83%

threw the bucket of acid on the experimenter when told to!

There are various social roles subjects adopt during a study. The

good-subject role tries to comply with everything the

experimenter wants. The faithful-subject role tries to be honest

no matter what. The negativistic-subject role attempts to

sabotage the study (it's been called the "screw you effect"). The

apprehensive-subject role is nervous about being evaluated and

tries to perform in a socially desirable way. Attempting to control

5

these demand characteristics can be challenging, but in general the double-blind procedure is the

way to go. Some researchers go so far as to run a second experiment - a "simulated experiment"

in which you simply tell subjects to behave as if they were in the actual experiment. By

comparing the results of the real study to the simulated one, the experimenter can get a handle on

the extent to which demand characteristic may be confounding the results.

Culture and Society

Our culture dictates what is acceptable and what is not in

Psychological research. For example, one time it was

acceptable to use prisoners in research. This population is

mostly male and mostly black. External validity becomes

an obvious problem. A great deal of research is conducted

using soldiers in the military. Again, mostly male, and

now mostly poor and uneducated. Here at MTSU, we

typically use the subject pool. MTSU is predominantly

white and female! External validity becomes a major

issue.

Obviously, our culturally shared ideas about the world also influence

how we respond to questions in survey research and respond in

experiments to various manipulations. The nearly universal belief that

drugs are bad leads people to deny drug use making it virtually

impossible to assess the true level of drug use in high schools, colleges, on the job, etc. Our

belief that Psychologists are involved in deception research leads us to search for the deception

in the experiment, to try to figure out what is "really" going on.

The history of science is the history of

combating peoples beliefs. The world is flat,

the earth is the center of the universe, the

stars are holes through which heaven shines,

a divine creator is required to "make" life

happen, there is no such thing as global

warming, etc. Our notions about what is real

and what is not influences how we see thing,

how we react to things. My least favorite one

is the emphasis on genetics in today's culture.

It is difficult to explore methods of behavior

change when the subjects believe that the

behavior is genetically determined and

therefore cannot be changed. (Of course you

are not responsible for your behavior then

either! Ain't genetics wonderful!)